Speculative Decoding
An LLM inference acceleration technique formally proposed by the Google Brain team in 2023. It introduces a small draft model to rapidly generate candidate token sequences, which are then verified in parallel by the main model, theoretically improving generation speed by a factor of two to four.
Core Facts
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
A higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
NVIDIA's TensorRT-LLM deeply optimizes inference speed for its hardware architecture.
Timeline (last 90 days)
When evaluating the speedup of MoE speculative decoding, both the number of accepted tokens and the number of unique experts should be reported, to avoid the optimistic bias of a single metric.
The end-to-end speedup of MoE speculative decoding depends on two factors: the acceptance rate of draft tokens and the degree of expert overlap within the draft window.
In the ideal case of speculative decoding, if the draft proposes γ tokens at once and the target model accepts α of them, then each round generates α+1 tokens while paying the cost of only one target-model forward pass.
When evaluating a speculative decoding scheme, one should focus on the average acceptance length on one's own task distribution, the drafter's memory footprint, and the difference in time to first token with it enabled versus disabled, rather than looking only at the peak speedup.
Speculative decoding optimizes inter-token latency (ITL) in the decoding stage and offers essentially no improvement to time to first token (TTFT), which may even increase slightly due to loading the drafter.
The relative gains of speculative decoding in high-concurrency, large-batch scenarios are usually lower than in low-latency single-request scenarios, which is jointly determined by the parallelism of the verification stage and memory bandwidth.
In code scenarios, each verification step accepts about 7.4 tokens, reaching a throughput of 154 tok/s; in prose scenarios each step accepts about 2.5 tokens, with a throughput of 35-37 tok/s; the ordinary decoding baseline throughput is about 18-19 tok/s.
Code-type tasks can achieve a throughput improvement of about 4x or more (154 vs 18-19 tok/s), while prose-type tasks achieve only about 2x (35-37 vs 18-19 tok/s).
In speculative decoding, the block length k should match the effective branching factor; too large a k amplifies the cost of mismatch.
Cross-verification from multiple sources indicates that the low prose acceptance rate is a characteristic of the drafter itself, not a problem with a particular runtime or hardware.
40 more timeline events
All Facts (20)
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedA higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
85%VerifiedNVIDIA's TensorRT-LLM deeply optimizes inference speed for its hardware architecture.
85%VerifiedSpeculative decoding typically delivers a 2–3× speedup in practical applications.
80%VerifiedMTP is a technique that allows the model to predict multiple subsequent tokens in a single forward pass, reducing the number of inference computation rounds.
80%VerifiedSpeculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
75%VerifiedInference throughput optimization relies on technologies such as KV Cache management and Continuous Batching, which directly determine response speed and service cost under concurrent user loads.
75%VerifiedSpeculative decoding is a lossless acceleration technique that does not alter the model's output distribution; it uses a draft model with far fewer parameters than the target model to generate candidate tokens, which the target model then verifies in parallel.
75%VerifiedSpeculative decoding submits the candidate sequence generated by the draft model to the target model as a whole for one-shot batch verification, requiring only a single forward pass to determine whether each candidate token is correct.
65%VerifiedSpeculative decoding's rejection-sampling-based verification algorithm ensures that the output distribution is exactly equivalent to—rather than merely an approximation of—running the main model alone.
65%VerifiedThe combination of Multi-Token Prediction and speculative decoding can achieve a 2–4x inference speed improvement without compromising generation quality.
65%Verified推测解码（Speculative Decoding）是目前最主流的大模型推理加速路径之一，使用小型草稿模型快速生成候选token再由主模型并行验证
65%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%UnverifiedA reasonable prediction loss combined with 0% downstream control accuracy is highly consistent with representation collapse, meaning the encoder has fooled the self-supervised objective without learning any useful dynamic information.
70%UnverifiedSpeculative decoding generates drafts with a small model and verifies them in batch with a large model; mathematically it guarantees output fully consistent with the large model through rejection sampling, making it lossless acceleration
70%UnverifiedAccording to DeepSeek's data, confidence scheduling in speculative decoding yields about a 6% throughput improvement.
60%UnverifiedWhen evaluating the speedup of MoE speculative decoding, both the number of accepted tokens and the number of unique experts should be reported, to avoid the optimistic bias of a single metric.
50%UnverifiedThe end-to-end speedup of MoE speculative decoding depends on two factors: the acceptance rate of draft tokens and the degree of expert overlap within the draft window.
50%UnverifiedThe relative gains of speculative decoding in high-concurrency, large-batch scenarios are usually lower than in low-latency single-request scenarios, which is jointly determined by the parallelism of the verification stage and memory bandwidth.
50%UnverifiedSpeculative decoding optimizes inter-token latency (ITL) in the decoding stage and offers essentially no improvement to time to first token (TTFT), which may even increase slightly due to loading the drafter.
50%