Verified75% confidenceFactExact time
When generating each token, a Transformer model needs to compute the Key and Value vectors of all previous tokens, and these tensors are collectively called the KV cache.
3
Sources
75%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedKV Cache is a key data structure in autoregressive generation by large models: when generating each new token the model needs the Key and Value vectors of all previous tokens, and if these were recalculated every time the cost would grow quadratically with sequence length.79% similarUnverifiedDuring Transformer model inference, input tokens can be processed in parallel (prefill stage), while output tokens must be generated one by one autoregressively (decode stage), causing the unit price of output tokens to typically be higher than that of input tokens.78% similarUnverifiedThe context cache (KV Cache) is the memory area that stores the Key and Value vectors of previous tokens during inference of Transformer-type models, and its size grows linearly with context length, batch size, and the number of model layers.75% similarVerifiedThe KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.74% similarUnverifiedPrefix cache reuse causes KV values to be computed in a different batch than a fresh prefill, so the KV values from a cache hit may differ at the floating-point level from those of a fresh prefill, and the difference propagates through subsequent tokens.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1031674API
curl https://kongchang.com/api/v1/knowledge/claims/1031674MCP
get_claim(id=1031674)