Speculative Decoding
A large language model inference acceleration technique in which a small draft model rapidly generates candidate token sequences, which are then verified in parallel by the larger target model. This approach increases generation speed while maintaining output quality.
Core Facts
Timeline (last 90 days)
In speculative sampling, when the acceptance rate of candidate tokens is high, the effective number of output tokens per forward pass can reach 2–4, thereby improving the overall generation speed by 1.5x–3x.
All Facts (3)
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%UnverifiedIn speculative sampling, when the acceptance rate of candidate tokens is high, the effective number of output tokens per forward pass can reach 2–4, thereby improving the overall generation speed by 1.5x–3x.
50%Unverified投机采样通过小型草稿模型生成候选token由大模型并行验证，可在不损失质量前提下将吞吐提升2-3倍
50%