Verified80% confidenceFactExact time
The VRAM usage of KV Cache grows linearly with sequence length; for a 70-billion-parameter model using GQA, the KV Cache usage can reach tens of GB when processing 100,000 tokens.
6
Sources
80%
Confidence
Long-term
Relevance
7/12/2026
First Seen
Sources
Related Claims
VerifiedThe VRAM occupied by KV Cache grows linearly with sequence length.82% similarVerifiedThe VRAM consumption of the KV cache is proportional to context length, number of model layers, and number of attention heads; under an 8K context, a typical 27B model may consume an additional 2–4 GB of VRAM.78% similarUnverifiedFor the LLaMA-2 70B model, when processing a context of 4096 tokens, the KV Cache requires approximately 20 GB of system memory at FP16 precision.78% similarUnverifiedAccurately estimating a model's VRAM usage requires considering multiple dimensions including KV Cache, inference framework runtime overhead (typically an additional 0.5–2 GB), and activation values.77% similarVerifiedWhen the context length is extended to one million tokens, the KV Cache itself can occupy tens of gigabytes of VRAM.76% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/489057API
curl https://kongchang.com/api/v1/knowledge/claims/489057MCP
get_claim(id=489057)