Context Caching
The practice of pre-storing recurring system prompts or long documents so that subsequent requests can reuse them at a lower unit price. It suits multi-turn question answering or fixed knowledge base scenarios and can reduce token costs.
Core Facts
Whether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.
Conventional KV caching is an optimization within a single request: within the same conversation round, the KV generated from historical messages does not need to be recomputed, and only the newly input content is computed incrementally.
Token caching (Prompt Caching) is a cost-optimization mechanism in large model APIs; when the same prefix appears repeatedly across multiple requests, the computation result can be cached and charged at a lower rate.
Timeline (last 90 days)
DeepInfer's core mechanism is context caching, which keeps already-computed mathematical paths in memory, with a cache validity period exceeding 72 hours.
Gemini API context caching comes in two forms: implicit caching (the system automatically identifies repeated prefixes) and explicit caching (developers actively create cache objects and specify a TTL).
The context cache is fully readable before it expires and may be captured by logging systems or observability tools; revoking an API key does not trigger cache invalidation.
When processing a long document of 8000 tokens, the KV cache may consume several additional GB of memory.
Whether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.
According to Anthropic's published prices (input $10, output $50, cache read $1, cache write $12.50 per million tokens), the example session in the article costs $16.88, of which cache reads account for 29.6%, cache writes 25.9%, fresh input 14.8%, and output 29.6%.
Conventional KV caching is an optimization within a single request: within the same conversation round, the KV generated from historical messages does not need to be recomputed, and only the newly input content is computed incrementally.
Token caching (Prompt Caching) is a cost-optimization mechanism in large model APIs; when the same prefix appears repeatedly across multiple requests, the computation result can be cached and charged at a lower rate.
All Facts (7)
Whether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.
70%VerifiedConventional KV caching is an optimization within a single request: within the same conversation round, the KV generated from historical messages does not need to be recomputed, and only the newly input content is computed incrementally.
65%VerifiedToken caching (Prompt Caching) is a cost-optimization mechanism in large model APIs; when the same prefix appears repeatedly across multiple requests, the computation result can be cached and charged at a lower rate.
65%UnverifiedDeepInfer's core mechanism is context caching, which keeps already-computed mathematical paths in memory, with a cache validity period exceeding 72 hours.
50%UnverifiedGemini API context caching comes in two forms: implicit caching (the system automatically identifies repeated prefixes) and explicit caching (developers actively create cache objects and specify a TTL).
50%UnverifiedThe context cache is fully readable before it expires and may be captured by logging systems or observability tools; revoking an API key does not trigger cache invalidation.
50%UnverifiedWhen processing a long document of 8000 tokens, the KV cache may consume several additional GB of memory.
50%