Unverified50% confidenceSolutionExact time
Approximately 200 GB of n-gram tables are stored in regular system memory rather than on the GPU, with background transfer prefetching the rows about to be used.
1
Sources
50%
Confidence
Long-term
Relevance
9/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor a typical 70-billion-parameter model, the KV cache occupies approximately 0.5 MB of memory per token, and generating 100,000 tokens requires carrying 50 GB of cache.71% similarVerifiedThe KV Cache of a 70B-parameter model with 128K context support can require hundreds of gigabytes of VRAM at full load, exceeding the 80GB VRAM capacity of a single H100.70% similarUnverifiedAt the 100 million token scale, the archive occupies 28.8 GB plus a 2.2 GB index on disk, while in-process memory via memory-map is only approximately 28 MB.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/923875API
curl https://kongchang.com/api/v1/knowledge/claims/923875MCP
get_claim(id=923875)