MTP
An inference acceleration technique that predicts multiple future tokens in a single forward pass and uses a verification mechanism to determine which predictions are accepted, substantially increasing token generation speed. The approach is conceptually related to Speculative Decoding.
Core Facts
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
Multi-Token Prediction (MTP) is a training and inference acceleration technique proposed by Meta in 2024 and applied in models such as DeepSeek-V3.
Timeline (last 90 days)
Using the space freed up by speech quantization, the developer raised the context from 32K to 100K and added MTP (for accelerated tool-call prediction) and a vision encoder.
For scenarios that pursue whole-passage speed (such as short question answering, writing, and code completion), MTP should be enabled, because short question answering is 27% faster and long documents are 15% faster.
MTP is not an external plugin or a switch that requires additional parameter tuning; instead, whether it is enabled is distinguished by whether the model filename contains the three letters "MTP."
The benefit of MTP is highly scenario-dependent: the shorter the input, the lighter the prefill and the higher the decoding proportion, making the decoding time saved by MTP more noticeable; the longer the input, the heavier the prefill and the more easily MTP's extra overhead is amplified.
MTP achieves acceleration by simultaneously optimizing the decoding phase (guessing multiple words at once, reducing the number of forward passes) and changing how prefill and decoding connect; the longer first-token latency in long-document scenarios is because the prefill phase needs extra processing for the guessing logic.
In the short question-answering test (a 400-character input plus a small question), the version with MTP reduced first-token latency from 5.9 seconds to 4.3 seconds, increased output speed from 57.98 characters/second to 66.14 characters/second (about 14% faster), and shortened the total time to finish the answer from 85.9 seconds to 62.3 seconds (27% faster).
For scenarios where only the opening paragraph is needed and you do not wait for the entire passage to be written, MTP should be disabled, because waiting an extra 6 seconds is a pure loss.
Among currently public measured data, the acceptance rates of drafters such as EAGLE, block diffusion, or MTP on open-domain prose under greedy decoding mostly cluster in the 2.5-2.6 range, and no report of a stable breakthrough beyond 3.5 has been seen yet.
A user reports a prose acceptance rate of 2.62 when using the MTP (Multi-Token Prediction) method.
On the same benchmark with the Qwen3.8-27B model, DFlash2 achieves a 2.80x speedup, DSpark 2.57x, and MTP 2.22x.
40 more timeline events
All Facts (20)
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
90%VerifiedSpeculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedMulti-Token Prediction (MTP) is a training and inference acceleration technique proposed by Meta in 2024 and applied in models such as DeepSeek-V3.
85%VerifiedMTP is a technique that allows the model to predict multiple subsequent tokens in a single forward pass, reducing the number of inference computation rounds.
80%VerifiedMTP adds multiple prediction heads to the model's output layer, enabling the model to simultaneously predict the probability distribution of the next N tokens in a single forward pass, achieving 2–4x inference acceleration without sacrificing generation quality.
80%VerifiedThe actual speedup gained from MTP depends on the prediction acceptance rate; if predictions are frequently incorrect and fall back to token-by-token decoding, the acceleration benefit is significantly reduced.
75%VerifiedDeepSeek's MTP base-version draft model is co-optimized during main model training to align the draft model's output distribution with the target model, distinguishing it from post-hoc distillation approaches.
75%VerifiedDeepSeek introduced MTP technology as early as version V3.
75%VerifiedCompared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
65%VerifiedMTP (Multi-Token Prediction) is a configuration flag that is disabled by default in most mainstream inference tools and can accelerate inference.
65%VerifiedMTP (Multi-Token Prediction) technology was systematically proposed by Meta in 2024 research and has been adopted by mainstream open-source models such as Qwen and DeepSeek.
65%UnverifiedMTP一次前向传播可同时产出多个token：主头输出token+1，第一个MTP头输出token+2，第二个MTP头输出token+3
90%UnverifiedMTP的额外预测头直接复用Transformer的隐藏状态表示，因此额外的计算开销极小
90%UnverifiedMTP（多令牌预测）最初由Meta在研究论文《Better & Faster Large Language Models via Multi-token Prediction》中系统提出
85%UnverifiedMTP与DFlash在架构上有本质不同，但都属于投机解码技术谱系
85%Unverified加载MTP层后额外浮点运算会导致数值产生微小变化，可能导致模型走不同推理路径产生不同结果，即使温度设为0也会出现
85%UnverifiedGoogle在Gemma 4中实际应用了MTP技术
80%UnverifiedMeta在LLaMA 3的技术报告中首次将MTP作为训练目标引入
80%UnverifiedMTP和Speculative Decoding两项技术理论上可以叠加使用
75%UnverifiedThrough MTP (Multi-Token Prediction) technology, the inference speed of Qwen3.8-Flash-Next and GLM-5.3-Flash is improved by up to 2x.
70%