Speculative Decoding
An LLM inference acceleration technique formally proposed by Google DeepMind in 2023. A smaller draft model rapidly generates candidate tokens, which a larger target model then verifies in parallel. This approach can significantly reduce inference latency and computational cost.
Core Facts
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
A higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
Timeline (last 90 days)
dSpark speculative decoding adds a "confidence head" to the drafter, truncating low-confidence candidates early, improving cluster speed by 4% to 7% without sacrificing accuracy.
For speculative decoding to work, the draft and target must be aligned in vocabulary and output space, so the draft model often needs to reuse the target's output head or have its own matching output head.
The verification stage of speculative decoding can be executed in batches, typically improving overall throughput by 1.5–3x, and is theoretically equivalent to the original model's greedy/sampling results.
According to DeepSeek's data, confidence scheduling in speculative decoding yields about a 6% throughput improvement.
In speculative decoding, more drafts are not always better; if subsequent drafts often fail to pass, both generation and verification waste compute.
Under the MoE architecture, differences in expert routing between the speculative decoding draft model and the large model bring additional challenges, requiring targeted adaptation by the engine.
Traditional speculative decoding relies on an independent, smaller draft model to serially generate candidate tokens, which are then verified in parallel by the large model.
In theory, if γ tokens are generated per draft and the acceptance length is τ, the ideal speedup is approximately τ.
The speedup benefit of speculative decoding depends on the product of two factors: the draft acceptance rate and the average length of accepted tokens.
Speculative decoding technology makes the speed approximately 1.6 to 1.8 times faster, and in actual tests the large model accepted the small assistant's guesses approximately 79% of the time
32 more timeline events
All Facts (20)
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
90%VerifiedSpeculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedA higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
85%VerifiedSpeculative decoding typically delivers a 2–3× speedup in practical applications.
80%VerifiedSpeculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
75%VerifiedWhen speed improvements and price reductions occur simultaneously, they typically point to underlying inference architecture optimization, with possible paths including model distillation, quantization, speculative decoding, and streaming processing architectures.
75%VerifiedSpeculative decoding is a lossless acceleration technique that does not alter the model's output distribution; it uses a draft model with far fewer parameters than the target model to generate candidate tokens, which the target model then verifies in parallel.
75%VerifiedSpeculative decoding was formally proposed by Google DeepMind in 2023; overall throughput can be improved by 2–3 times, and the output quality is completely equivalent to that of the original large model.
75%VerifiedCompared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
65%VerifiedSpeculative decoding submits the candidate sequence generated by the draft model to the target model as a whole for one-shot batch verification, requiring only a single forward pass to determine whether each candidate token is correct.
65%VerifiedSpeculative decoding was formally proposed by Google in 2023, using a draft model with 1/10 to 1/50 the parameters of the target model to generate multiple candidate tokens at once, which are then verified in parallel by the large model.
65%VerifiedSpeculative decoding uses a small local model to rapidly generate draft tokens, which are then verified and corrected by a large cloud-based model.
65%VerifiedThe combination of Multi-Token Prediction and speculative decoding can achieve a 2–4x inference speed improvement without compromising generation quality.
65%Verified推测解码（Speculative Decoding）是目前最主流的大模型推理加速路径之一，使用小型草稿模型快速生成候选token再由主模型并行验证
65%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%Unverified推测解码技术已被Google、Meta等公司广泛应用于生产环境
80%UnverifiedSpeculative decoding is a lossless LLM inference acceleration technique formally proposed by Leviathan et al. in 2023.
70%UnverifiedThe verification process in traditional speculative decoding accepts draft-proposed tokens with probability min(1, p(x)/q(x)), and upon rejection resamples from the corrected distribution norm(max(0, p(x)-q(x))), which can be proven to produce a token distribution identical to directly sampling from the target model (lossless).
70%UnverifiedAccording to DeepSeek's data, confidence scheduling in speculative decoding yields about a 6% throughput improvement.
60%UnverifiedLing 3.0 Tiny achieves an inference speed of approximately 86–90 tokens per second on a MacBook with M4 Pro, with a peak memory usage of approximately 8.34 GiB under an 8K context.
60%