Verified75% confidenceFactExact time
During autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.
3
Sources
75%
Confidence
Long-term
Relevance
8/26/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe autoregressive generation process of large language models requires frequent reading of the KV Cache (key-value cache), placing extremely high demands on memory bandwidth and capacity.78% similarUnverifiedWhen large models generate long texts, they need to cache the key-value pairs (KV Cache) of all previous tokens, and the memory footprint grows linearly with context length, making it very easy for this to become a memory bottleneck in high-concurrency scenarios.76% similarVerifiedKV Cache stores the Key and Value matrices of all previous tokens during the Decode phase, grows with conversation turns and context window size, and can occupy anywhere from a few GB to tens of GB of memory in long-text scenarios75% similarUnverifiedGeneration throughput (tok/s) reflects decoding-phase efficiency and is affected by KV Cache size and memory bandwidth, with far lower sensitivity to input length than prefilling.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/802107API
curl https://kongchang.com/api/v1/knowledge/claims/802107MCP
get_claim(id=802107)