LLaDA
A large language model based on diffusion mechanisms that generates text through iterative denoising, in contrast to the mainstream autoregressive LLM approach. Instead of predicting tokens strictly left to right, it starts from a noise sequence and progressively recovers the text over multiple rounds. This route is studied for its potential advantages in global information use and parallel generation.
Timeline (last 90 days)
Diffusion language models (such as LLaDA and Dream) use iterative masked decoding, starting from a fully masked sequence and progressively revealing tokens over several denoising steps, with each step applying bidirectional attention over the entire sequence, so the representations of already revealed tokens are updated by newly revealed tokens.
On the LLaDA and Dream models, evaluated on four benchmarks—GSM8K, HumanEval, MATH, and MBPP—dLLM-KVQ achieved the highest average score among all quantization methods in the reported 4-bit and 2-bit prefix/dual cache settings.
Unsloth now supports training and chatting with text diffusion language models such as LLaDA, Dream, SDAR, LLaDA2, Nemotron-Labs-Diffusion and DiffusionGemma.
Unsloth v0.1.906 adds DiffusionTrainer, which can train text diffusion language models such as DiffusionGemma, LLaDA, LLaDA2, Dream, Nemotron and SDAR.
All Facts (4)
Diffusion language models (such as LLaDA and Dream) use iterative masked decoding, starting from a fully masked sequence and progressively revealing tokens over several denoising steps, with each step applying bidirectional attention over the entire sequence, so the representations of already revealed tokens are updated by newly revealed tokens.
50%UnverifiedOn the LLaDA and Dream models, evaluated on four benchmarks—GSM8K, HumanEval, MATH, and MBPP—dLLM-KVQ achieved the highest average score among all quantization methods in the reported 4-bit and 2-bit prefix/dual cache settings.
50%UnverifiedUnsloth now supports training and chatting with text diffusion language models such as LLaDA, Dream, SDAR, LLaDA2, Nemotron-Labs-Diffusion and DiffusionGemma.
50%UnverifiedUnsloth v0.1.906 adds DiffusionTrainer, which can train text diffusion language models such as DiffusionGemma, LLaDA, LLaDA2, Dream, Nemotron and SDAR.
50%