KV Cache
KV Cache (Key-Value Cache) is an optimization mechanism used in large language model inference to store the key and value tensors computed during the attention calculation process. By reusing previously computed results, it avoids redundant computation over already-processed tokens, significantly reducing computational overhead during autoregressive generation and improving inference efficiency. KV Cache is widely applied in the deployment and acceleration of Transformer-based models.
Core Facts
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
PagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
Timeline (last 90 days)
DCKV is a differentiable KV cache codec that, for the first time, enables end-to-end training, jointly optimizing the transform, quantizer and entropy model with a frozen language model.
When long prompts extend request residency time, if batch slot reuse lacks strict session-boundary cleanup, residual KV state may be read by the next request.
Long prompts triggering cross-user leakage is usually related to how the inference engine manages context length, batching strategy, and KV cache.
RadixAttention prefix caching builds the prompts of multiple requests into a radix tree, so identical prefixes compute the KV cache only once and subsequent requests can reuse it directly.
The sharing granularity of large-model inference is finer than that of traditional SaaS—not only are compute resources shared, but also intermediate computation results (KV cache) and batch slots.
If the cache is not properly isolated between requests, one user's context information may leak to another user, resulting in cross-request data exposure.
In multi-tenant mode, the isolation boundary extends not only to networking and storage but also to intermediate state in GPU memory, and the KV cache is the most easily overlooked part of this.
Over the past year, research on KV cache compression, paged management and prefix reuse has emerged densely, and long-context cost has become one of the core bottlenecks for agent deployment.
The degree to which cache can be discarded depends on how strongly a task relies on historical information: information-retrieval tasks may benefit more, while tasks requiring strict tracking of early instructions may not benefit.
The framework uses a distributed file system to build a KV cache pool with a capacity of 100,000 TB, retaining memory for three days for agent scenarios and seven days for human chat scenarios.
40 more timeline events
All Facts (20)
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
90%VerifiedKV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedKV Cache is the persistent storage of intermediate states in the Transformer attention mechanism, recording the key and value vectors for each attention head's processed tokens across all layers, thereby skipping redundant forward propagation computations.
80%VerifiedvLLM's PagedAttention draws on the operating system concept of virtual memory paging to manage KV caches, reducing GPU memory fragmentation overhead from approximately 30% to near 0.
80%Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%VerifiedQuantized KV caches (such as storing KV vectors in FP8 or INT4) can compress the cache size to 1/2 or even 1/4 of the original.
80%VerifiedThe KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.
80%VerifiedFor a model with 70B parameters, the KV Cache for an 8K context may occupy several additional gigabytes of VRAM
80%VerifiedKV cache size grows linearly with context length, and attention computation grows quadratically with context length.
75%VerifiedThe longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
75%VerifiedThe cost of persistent KV caching is a significant increase in storage space, along with the engineering challenge of efficiently locating and loading relevant segments from a massive cache.
75%VerifiedThe KV cache grows linearly with context length, increasing the memory read volume per step in the decoding phase, which is the fundamental reason decoding speed decreases as context grows.
75%VerifiedThe higher the cache hit rate, the closer the actual billing is to the cache price rather than the full input price, which can reduce costs by an order of magnitude.
75%VerifiedKVCache is a mechanism in the Transformer architecture that caches the Key and Value vectors of all previous tokens to avoid redundant computation, with its memory footprint growing linearly with context length.
75%VerifiedCompared to the previous-generation V4 Flash, V4.1 Flash reduces the demand for high-bandwidth memory (HBM) to one-quarter and the demand for SSD persistent storage to one-eighth.
75%