Masked Diffusion Model

Generative modeling of discrete data has long been dominated by two paradigms: GPT-style autoregressive models (generating token by token from left to right) and masked diffusion models (generating in arbitrary order).

Anari et al.'s scheme requires one score query for every token unmasked, so the number of queries grows linearly with the block size.

SIRS (Single-Increment Rejection Sampler) randomly samples one increment and uses a single query to estimate the log acceptance probability of the entire block, thereby reducing query cost while preserving unbiasedness.

The noising process of a masked diffusion model (MDM) progressively replaces tokens in the sequence with a special [MASK] token, and the training objective is for the model to learn to predict the original token at each masked position based on the currently partially visible sequence.

During MDM generation, the conditional distribution at each position can be computed simultaneously, which naturally supports parallel decoding, in sharp contrast to the way autoregressive models generate tokens one by one.

Anari et al. (2026) proposed using rejection sampling to correct the distributional bias in parallel decoding of masked diffusion models, decomposing the log acceptance probability into a sum of token-level terms.

Masked diffusion models can decode in parallel, support generation in arbitrary order, and are naturally suited to handling discrete representations of multiple modalities.

The latest research proposes the first trimodal masked diffusion model pretrained from scratch, trained simultaneously on text, image-text, and audio-text data.

Discrete diffusion models construct complete sequences by iteratively and in parallel resolving multiple tokens across multiple steps, unlike the left-to-right word-by-word generation of autoregressive models.