Diffusion Language Model

Diffusion language models adopt a parallel denoising approach, which can simultaneously optimize outputs at multiple positions in a single forward pass and then iteratively converge to the final text through multiple rounds.

The denoising process of diffusion models typically requires tens to hundreds of iterations (called sampling steps), each of which requires a complete forward pass through the neural network.

Diffusion language models can simultaneously perform parallel prediction and iterative refinement on all tokens in the entire sequence, rather than outputting them one by one.

Diffusion language models are still at an early stage and continue to lag behind mature autoregressive models in generation quality, long-text coherence, and training stability.

Diffusion language models still need to process all sequence positions at each denoising step, so parallel generation does not mean the end of computational waste.

Diffusion language models generate multiple tokens simultaneously through a parallel denoising process, promising to substantially reduce inference time.

Unlike BERT, which usually makes only one prediction, a dLLM starts from full masking or a high masking rate and iterates repeatedly, gradually reducing the mask until the entire sentence is filled in.

At each step, which positions a dLLM chooses to unmask first determines the prediction difficulty of each subsequent step.

Research from VILA-Lab proposes dSpark, a framework based on schedule distillation for optimizing the decoding order of diffusion language models.

The backbone of a diffusion language model (dLLM) is highly similar to that of a BERT-style masked language model: during training, a portion of the tokens is randomly masked, and the model restores them based on the remaining context.

Researchers report that NCM's generation speed on specific benchmarks can reach 31 times that of a diffusion language model baseline.

NCM is similar to diffusion models in that both abandon strict left-to-right token-by-token generation; the difference is that NCM organizes the generation process with an explicit multi-granularity discrete codebook and a clearer hierarchical structure, whereas diffusion models implicitly control generation granularity through noise levels.

The design of the training-time recurrence count distribution is an underappreciated lever in efficiency optimization for diffusion language models.

The reasoning capability of diffusion language models can be enhanced through inference-time search strategies without relying on additional post-training.

Diffusion language models have historically had shortcomings in factual accuracy and logical consistency over long texts.

Traditional diffusion language models often require dozens or even hundreds of denoising steps, making their actual inference cost higher than that of autoregressive models.

The community's attitude toward diffusion language models can be summed up as 'optimistic about specific tasks, skeptical about general reasoning'.

Each round of the multi-round denoising process of diffusion language models can process all positions in parallel, giving it a potential advantage in hardware utilization efficiency.

The trade-off between training efficiency and inference quality in diffusion language models is not yet ideal; many of the results shown depend on carefully designed experimental conditions, and a gap remains in performance on open-domain tasks.

Effective guidance of DLM depends on the compatibility between the content of the control token and its placement position; mismatched anchors interfere with generation capability.