Verified80% confidenceBenchmarkExact time
Taking a 70B parameter model with FP16 precision and 32 attention head layers as an example, the KV Cache for a single 200K-length request requires approximately 80–120 GB of VRAM, far exceeding the 80 GB VRAM limit of a single A100 GPU.
4
Sources
80%
Confidence
Long-term
Relevance
7/16/2026
First Seen
Sources
Related Claims
Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB79% similarVerifiedA GPT-4-class model with 175 billion parameters requires approximately 175 GB of VRAM even when stored at fp8 precision.78% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.78% similarUnverifiedA 763B-parameter model requires about 760GB or more of VRAM or RAM at 8-bit precision, and on the order of about 1.5TB at BF16 precision, far exceeding a single mainstream server.77% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/526414API
curl https://kongchang.com/api/v1/knowledge/claims/526414MCP
get_claim(id=526414)