Prefix Caching
An LLM inference optimization technique that persistently caches the KV computation results of shared prefix content (such as system prompts) across multiple requests, enabling cross-request reuse to reduce redundant computation, lower latency, and improve throughput.
Timeline (last 90 days)
Some inference frameworks support multi-level prefix tree matching, which can hit common ancestor nodes shared across different requests, thereby increasing the prefix sharing rate.
A common engineering practice for dealing with prefix invalidation is to move dynamic content (user input, current time, retrieval results) after the static prefix, forming a request structure of "stable prefix + variable suffix".
Prefix caching depends on strict conditions: the reused portion must be identical from the very first token of the sequence, otherwise the cache is invalidated and the entire segment is recomputed.
The cost of the fixed context can be approximated as: number of fixed-prefix tokens × number of calls × cache-read unit price.
Prefix Caching enables KV reuse across requests; vLLM uses block hashing, while SGLang uses a radix tree.
PagedAttention supports multiple requests sharing the KV Cache of the same prefix (Prefix Caching).
vLLM uses Prefix Caching technology to reuse the KV Cache of requests with identical prefixes.
There is a bug in the current vLLM implementation: in prefix caching, the last cache block of approximately 16 tokens is directly discarded and recomputed, even when a request hits the cache.
Prefix caching can save 50%–80% of prefill-stage computation by reusing already-computed KV Cache, and vLLM implements fine-grained cache hit matching through a radix tree structure.
When prefix caching and DFlash2 speculative decoding are both enabled simultaneously, total throughput drops by approximately 37%, which is even slower than not enabling speculative decoding at all.
All Facts (8)
Prefix caching depends on strict conditions: the reused portion must be identical from the very first token of the sequence, otherwise the cache is invalidated and the entire segment is recomputed.
50%UnverifiedSome inference frameworks support multi-level prefix tree matching, which can hit common ancestor nodes shared across different requests, thereby increasing the prefix sharing rate.
50%UnverifiedA common engineering practice for dealing with prefix invalidation is to move dynamic content (user input, current time, retrieval results) after the static prefix, forming a request structure of "stable prefix + variable suffix".
50%UnverifiedThe cost of the fixed context can be approximated as: number of fixed-prefix tokens × number of calls × cache-read unit price.
50%UnverifiedPrefix Caching enables KV reuse across requests; vLLM uses block hashing, while SGLang uses a radix tree.
50%UnverifiedPagedAttention supports multiple requests sharing the KV Cache of the same prefix (Prefix Caching).
50%UnverifiedvLLM uses Prefix Caching technology to reuse the KV Cache of requests with identical prefixes.
50%UnverifiedPrefix caching can save 50%–80% of prefill-stage computation by reusing already-computed KV Cache, and vLLM implements fine-grained cache hit matching through a radix tree structure.
50%