KV Cache
KV Cache (Key-Value Cache) is an optimization mechanism used during Transformer model inference to store the key and value vectors corresponding to each token in the attention computation. During autoregressive generation, the model reuses cached key-value pairs from previous tokens rather than recomputing them at each new generation step, significantly reducing redundant computation and improving inference efficiency. The mechanism trades increased GPU memory usage for faster inference speed and is a core optimization technique in large language model deployment.
Core Facts
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
The VRAM occupied by KV Cache grows linearly with sequence length.
Timeline (last 90 days)
KV Cache size is proportional to sequence length, number of layers, number of attention heads and batch size, and in long-context scenarios it may exceed the model weights themselves.
When using vLLM as the rollout engine, one needs to handle issues such as hot weight updates, KV Cache cleanup, scheduling priority, and coordination with the training process.
The industry typically mitigates the computation and memory issues of long sequences with techniques such as FlashAttention, sparse attention, sliding windows, chunked processing, and KV cache compression.
Service providers need to build a unified compute abstraction layer (similar to compiler infrastructure such as MLIR and TVM), and implement optimizations including operator fusion, continuous batching, and KV Cache management.
When traditional OCR models process long documents, memory and VRAM usage grow linearly with the number of pages.
The prefill stage is the process in which the model builds a key-value cache (KV Cache) for all input tokens when processing user input.
Hy-Nomas uses KV Cache as the memory mechanism for video context to maintain scene coherence.
Input cache hit rate refers to the proportion of input tokens in a request that hit the already-computed and cached KV Cache; a high hit rate can significantly reduce GPU compute consumption, but it depends on the request pattern and the degree of prefix reuse, and not all business scenarios can achieve it.
Long contexts significantly increase the GPU memory footprint of the KV Cache, which is often mitigated in engineering through grouped-query attention (GQA), KV Cache quantization or paged management.
Cache Read is a billing and optimization mechanism in large language model APIs: when the prefix of a request is exactly identical to that of a previous request, the provider can directly reuse the already-computed key-value cache (KV Cache) without rerunning attention computation.
40 more timeline events
All Facts (20)
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
90%VerifiedKV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedThe VRAM occupied by KV Cache grows linearly with sequence length.
90%VerifiedWhen large language models process long contexts, the computational complexity of the attention mechanism grows proportionally to the square of the sequence length.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedKV Cache reduces inference complexity from O(n²) to O(n) by caching the already-computed Key/Value matrices.
90%VerifiedPagedAttention divides the KV cache into fixed-size pages allocated on demand, and can improve VRAM utilization by 2–4×.
85%VerifiedPagedAttention improves KV Cache memory utilization from below 40% to close to 100%.
85%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedKV Cache is the persistent storage of intermediate states in the Transformer attention mechanism, recording the key and value vectors for each attention head's processed tokens across all layers, thereby skipping redundant forward propagation computations.
80%VerifiedPaged Attention (PagedAttention) is the core technology of vLLM, managing KV cache in blocks, while Flash Attention optimizes the memory access pattern of attention computation.
80%VerifiedGrouped Query Attention (GQA) and Multi-Query Attention (MQA) reduce KV cache memory usage during inference by having multiple query heads share the same group of key/value heads, and are standard optimizations in models such as Llama 3 and Mistral.
80%VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.
80%VerifiedLLaMA-2-70B's KV Cache for a single request at a context length of 128K can occupy more than 160 GB of VRAM.
80%VerifiedWhen using FP16 precision, the KV Cache for LLaMA-2-7B occupies approximately 0.5 MB per token.
80%VerifiedThe VRAM usage of KV Cache grows linearly with sequence length; for a 70-billion-parameter model using GQA, the KV Cache usage can reach tens of GB when processing 100,000 tokens.
80%Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedTransformer架构的自注意力机制在处理长序列时面临二次方复杂度的计算开销，上下文长度翻倍则计算量增长四倍
80%VerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%