Autoregressive Decoding
Core Facts
The bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
自回归大模型推理瓶颈并非计算量而是内存带宽，推测解码通过拒绝采样保证输出分布与原始大模型完全一致
Autoregressive text generation is a memory-bandwidth-bound task, where memory bandwidth—not peak compute—is the true performance bottleneck.
Timeline (last 90 days)
In generative large models, a single forward pass usually produces only one token, and to obtain a complete output the result must be fed back into the input and called repeatedly, i.e., autoregressive decoding.
Frontier API providers price output tokens at 4 to 5 times input tokens because autoregressive decoding is limited by GPU high-bandwidth memory bandwidth.
In autoregressive decoding, the probability of each token is conditioned on all preceding tokens, with the mathematical form P(x_t | x_1, ..., x_{t-1}).
Memory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
Matrix multiplication is typically compute-bound in high-batch inference scenarios, while autoregressive decoding with a batch size of 1 is almost invariably bandwidth-bound.
A drawback of autoregressive decoding is that inference speed is linearly constrained by sequence length.
The bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
Splicing autoregressive text generation together with diffusion-based iterative denoising introduces a fundamental asymmetry in computational paradigms.
All Facts (11)
The bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
75%Verified自回归大模型推理瓶颈并非计算量而是内存带宽，推测解码通过拒绝采样保证输出分布与原始大模型完全一致
65%VerifiedAutoregressive text generation is a memory-bandwidth-bound task, where memory bandwidth—not peak compute—is the true performance bottleneck.
65%UnverifiedIn generative large models, a single forward pass usually produces only one token, and to obtain a complete output the result must be fed back into the input and called repeatedly, i.e., autoregressive decoding.
50%UnverifiedFrontier API providers price output tokens at 4 to 5 times input tokens because autoregressive decoding is limited by GPU high-bandwidth memory bandwidth.
50%UnverifiedIn autoregressive decoding, the probability of each token is conditioned on all preceding tokens, with the mathematical form P(x_t | x_1, ..., x_{t-1}).
50%UnverifiedMemory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
50%UnverifiedMatrix multiplication is typically compute-bound in high-batch inference scenarios, while autoregressive decoding with a batch size of 1 is almost invariably bandwidth-bound.
50%UnverifiedA drawback of autoregressive decoding is that inference speed is linearly constrained by sequence length.
50%UnverifiedSplicing autoregressive text generation together with diffusion-based iterative denoising introduces a fundamental asymmetry in computational paradigms.
50%Unverified自回归解码具有串行特性，每个Token的生成依赖前一个Token，难以并行
50%