KV Cache

When generating each token, a Transformer model needs to compute the Key and Value vectors of all previous tokens, and these tensors are collectively called the KV cache.

In the KV Cache mechanism, reusing cached KV values is a cache hit, while context changes that invalidate the cache and require recomputation are a cache miss.

KV缓存将已计算的Key-Value矩阵存储在显存中实现增量计算，代价是显存占用随序列长度线性增长

The size of the KV cache grows linearly with context length, and VRAM consumption increases significantly when processing longer texts.

The precision loss of FP4 is more severe for data such as the KV cache that is sensitive to long-range dependencies, and it usually requires per-channel or per-token scaling strategies, as well as possibly keeping sensitive layers in higher precision.

According to DeepSeek's data, 96% of all input tokens hit the cache.

Most companies keep prompt caches for only five minutes, whereas DeepSeek extends cache duration to several days: three days for agent traffic and seven days for chat.

TTFT (time to first token) is mainly determined by the prefill stage, with computation roughly proportional to context length; prefix caching can reuse already-computed KV states and only prefill the newly added portion.

The GPU memory occupied by the KV cache grows with context length and dialogue turns, and is the main source of memory usage and latency in long-context deployment.

Model weights are only part of VRAM usage; during inference, space must also be reserved for activations, KV cache, context window, and so on.

Running the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).

Because expert pruning preserves the attention architecture and KV cache format, the KV cache handoff requires no cache conversion, and the runtime only needs to switch weight masks between the two stages.

The order in which prompts are organized directly affects the cache hit rate, and thus the actual bill.

For multi-turn conversations or long-document question answering, if the KV of the entire context is recomputed on every request, the compute overhead grows quadratically with context length.

Actual token generation speed usually reaches only 50%–70% of the theoretical value, affected by factors such as compute unit scheduling, KV cache reads and writes, and quantization-dequantization overhead.

The prefill phase processes the complete input context, computes attention for all input tokens, and generates the KV cache; it is a compute-intensive operation whose time consumption is strongly correlated with input length.

vLLM implements a bidirectional KV transfer mechanism in which decode nodes send computed KV blocks back to prefill nodes, eliminating repeated prefill computation of common prefixes in multi-turn conversations.

In an Agent session, multiple rounds of calls share the same context prefix, so in theory there are a large number of reusable KV cache blocks.