Verified65% confidenceFactExact time
The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.
3
Sources
65%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedLLaMA-2 7B's KV Cache occupies approximately 2 GB with a 4096-token context, and can surge to over 16 GB when extended to a 32K context78% similarUnverifiedFP8 KV cache is a quantization technique that compresses the Key-Value cache in the attention mechanism from FP16 or BF16 to 8 bits, which can halve the GPU memory usage of the KV cache77% similarUnverifiedFor a typical configuration (L layers, hidden dimension d, GQA with h KV heads), the KV Cache size per token is approximately 2 × L × h × d_head × number of bytes.77% similarUnverifiedFor the LLaMA-2 70B model, when processing a context of 4096 tokens, the KV Cache requires approximately 20 GB of system memory at FP16 precision.76% similarVerifiedThe size of the KV cache grows linearly with context length; the KV cache for Qwen 2.5 32B at a 64K context may consume several additional gigabytes of memory.76% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1041471API
curl https://kongchang.com/api/v1/knowledge/claims/1041471MCP
get_claim(id=1041471)