Unverified60% confidenceFactExact time
Generation throughput (tok/s) reflects decoding-phase efficiency and is affected by KV Cache size and memory bandwidth, with far lower sensitivity to input length than prefilling.
2
Sources
60%
Confidence
Long-term
Relevance
9/17/2026
First Seen
Sources
Related Entities
Related Claims
Unverified在多节点 prefill/decode 分离架构中，prefill 节点需将生成的 KV cache 通过高速网络传给 decode 节点，节点间通信是关键瓶颈，KV cache 传输效率直接决定分离架构能否兑现性能收益。79% similarVerifiedThe size of the KV cache grows linearly with context length, and VRAM consumption increases significantly when processing longer texts.77% similarUnverifiedLarge-model inference is a memory-bandwidth-intensive task: in the decode phase, all weights and the KV Cache must be re-read for each token generated, so the larger the batch size, the lower the marginal cost per token, but the higher the latency.76% similarUnverifiedMethods for compressing the KV Cache include quantization, low-rank approximation, and evicting old tokens by importance; their direct effect is to increase the number of tokens (capacity) that a unit of GPU memory can hold, rather than to save compute or transfer.75% similarUnverifiedMemory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/930282API
curl https://kongchang.com/api/v1/knowledge/claims/930282MCP
get_claim(id=930282)