Unverified50% confidenceFactExact time
When model weights and KV Cache cannot be fully loaded into GPU VRAM and some data must be offloaded to main memory, inference speed drops by one to two orders of magnitude.
1
Sources
50%
Confidence
Long-term
Relevance
9/10/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedGPU 推理速度通常是纯 CPU 的 5-20 倍，当模型太大无法放入显存时部分层会卸载到内存由 CPU 处理（混合运行/partial offload）80% similarUnverifiedThe core bottleneck in large model inference on GPUs is memory bandwidth; autoregressive generation of each token requires loading the full KV Cache from HBM, with compute utilization typically below 5%.78% similarUnverified自回归推理是memory-bound（内存带宽受限）问题，因为batch size通常为1时算术强度极低，GPU大部分时间在等待数据搬运76% similarUnverifiedModel weights cannot fill up VRAM entirely, as space must be reserved for the KV cache and inference processes; the KV cache required for a 128K context can reach as high as several GB or even exceed the model weights themselves.75% similarUnverified研究《Accelerating Block Low-Rank Foundation Model Inference on Memory-Constrained GPUs》提出基于块低秩分解的方案压缩内存并加速推理75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/888668API
curl https://kongchang.com/api/v1/knowledge/claims/888668MCP
get_claim(id=888668)