Constrained Decoding

Constrained decoding dynamically blocks candidate words that do not conform to the target grammar or schema when the model generates each token, ensuring the output structure is legal at the decoding level.

Constrained Decoding computes the set of valid tokens at each token step of autoregressive generation based on a finite state machine or context-free grammar and renormalizes the probabilities, mathematically guaranteeing that the output conforms to the target schema.

Constrained decoding forces the model to produce output conforming to a preset structure by masking out tokens that do not comply with the target grammar or schema at each generation step.

Constrained decoding can only guarantee that the syntactic structure of the output is legal (shape); it cannot guarantee the semantic correctness of the output content.

JSON Mode only guarantees that the output is valid JSON without constraining the field structure; Structured Outputs enforces output conforming to a predefined JSON Schema at the syntactic level by applying a finite state machine (FSM) to constrain the token sampling path during the decoding phase.

Constrained decoding restricts the model, via a predefined JSON Schema, so that each generated token can only be selected from the set of valid tokens.

Constrained Decoding guarantees at the sampling level that outputs will conform to a specified JSON Schema or regular expression; typical open-source implementations include Outlines, Guidance, and LM Format Enforcer.

The underlying implementation of constrained decoding relies on finite state machines (FSM) or context-free grammars (CFG).

The core method of grammar-constrained decoding is to use a context-free grammar (CFG) to filter out illegal token options during the model's token-by-token output generation, setting the logits of illegal tokens to negative infinity.

Constrained decoding and function calling can restrict the selectable tokens during the decoding stage, but they still belong to the generation paradigm and output text rather than a directly usable probability distribution.

Constrained decoding filters candidate tokens through finite state machines or grammar rules; if only one legal token remains after filtering, the sampling strategy ceases to have any effect and the output path is uniquely determined.

State drift refers to the phenomenon in which the model gradually deviates from the correct state over multi-step execution; constrained decoding guarantees that each step's output format is legal, but cannot guarantee semantic correctness.

Constrained decoding applies a mask to the vocabulary at each decoding step, setting the probability of tokens that violate constraints to zero; the mask computation itself has overhead, and complex grammars slow down decoding.

Constrained decoding, which restricts the set of candidate words during generation, is one of the mainstream engineering methods for suppressing hallucination today.

Instruction following and tool calling depend on strict structural constraints on the sequence (such as JSON syntax and function signatures), and the globally parallel updates of diffusion decoding do not inherently guarantee that these constraints are satisfied bit by bit, typically requiring additional constrained decoding or post-processing correction.

Constrained decoding dynamically masks, at each sampling step, tokens that do not conform to the target grammar (such as a regular expression or context-free grammar), mathematically excluding illegal outputs.

Common implementations of structured output include providing a function calling interface, constrained decoding, or post-processing parsing plus validation.

Constrained decoding masks tokens that do not conform to the grammar at the sampling stage, offering the highest reliability.

The cost of constrained decoding is a possible slight loss of generation quality, but it has almost no impact on short-output tasks such as classification and scoring.

Constrained decoding is a technique that forces the model's output to conform to a specific grammar or format during inference, masking to 0 the probability of tokens that do not conform to a predefined grammar (such as JSON Schema) at each token generation step.