Prefix Caching

The KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.

Prefix caching improves inference efficiency by caching common prompt prefixes to reduce redundant computation.

Caching means pre-storing recurring system prompts or long documents so that subsequent requests reuse them at a lower unit price, suitable for multi-turn Q&A or fixed knowledge base scenarios.

Mainstream inference frameworks such as vLLM already natively support automatic prefix caching.

在语言模型推理优化中，常见手段大致可分为两类：一类是降低单次推理的计算与显存开销（如量化、知识蒸馏、稀疏化与剪枝），另一类是减少重复计算、提升吞吐（如KV Cache复用、前缀缓存、连续批处理与投机解码）。

In multi-tenant scenarios, the prefix cache key must include a unique tenant or session identifier; otherwise identical prefixes from different users will hit the same cache entry, causing context cross-talk.

In DeepSeek's own traffic, 96% of inputs hit the prefix cache, and single-GPU prefill throughput jumped from 60,000 tokens per second to 1.5 million tokens per second, a 25x improvement.

Mastering the underlying principles of large models (such as KV cache and prefix cache), hybrid retrieval, model fine-tuning, reinforcement learning, data cleaning, and evaluation set construction qualifies one for a project lead role and positions benchmarked at 30K and above.

When the prefix cache is hit, repetitive prefill of tens of thousands of tokens can be almost entirely skipped, dropping first-token latency from seconds to milliseconds.

Prefix cache reuse causes KV values to be computed in a different batch than a fresh prefill, so the KV values from a cache hit may differ at the floating-point level from those of a fresh prefill, and the difference propagates through subsequent tokens.

After using the KV cache, single-token latency remains essentially flat and no longer grows significantly with context length, indicating that the bottleneck is no longer matrix multiplication but fixed overheads such as CPU-side scheduling, kernel launch overhead or GPU memory bandwidth.

Static system prompts enjoy a 90% KV cache discount, but dynamic chain-of-thought traces are stripped between turns or regenerated at each step, so prefix caching cannot amortize test-time compute bursts across multi-turn agent sessions.

Publishing determines when data can be reused, and coordination determines who waits and for how long.

In the self-developed Rust LLM inference engine mini-vllm-rs, after prefix caching is enabled, when two completely identical requests arrive simultaneously at a cold cache, possibly not a single token can be reused, and both perform full prefill.

Prefix caching splits the prompt into cache pages of a fixed size (e.g., 16 tokens), each page corresponding to a segment of KV, and builds a hash index; when a new request arrives, the input tokens are compared page by page, and a hit allows referencing the already-computed KV page and skipping the corresponding prefill computation.

The key constraint of prefix caching is that a cache page must only become visible to other requests after computation is complete and after it has been correctly published, and the timing of visibility directly determines whether a hit is possible in concurrent scenarios.

What local inference frameworks can do is optimize KV cache management (such as paged attention and prefix caching), so that long conversations are not interrupted by running out of VRAM, thereby remaining stable in multi-turn scenarios.

Production-grade AI infrastructure typically employs three layers of caching: the first layer is exact hash caching, the second layer is semantic caching, and the third layer is upstream provider and prefix caching.

Prefix caching is provided directly by model providers such as Anthropic and OpenAI, retaining the KV activation state of a static prompt prefix and reusing precomputed attention states, but it still incurs provider latency and network transmission overhead, with typical read latency of 200ms-1500ms.

Estimating loop cost at flat input prices, under near-ideal caching conditions it was overestimated by about 7 times on average.