Prompt Caching

The higher the cache hit rate, the closer the actual billing is to the cache price rather than the full input price, which can reduce costs by an order of magnitude.

The core principle of prompt caching is cross-request reuse of the KV Cache, where already-computed KV states can be cached and reused when multiple requests share the same prefix.

The KV Cache mechanism avoids repeated computation of historical tokens, but its memory usage grows linearly with context length.

Prompt caching means the identical prefix shared by multiple requests is fully billed and processed only on the first request, and subsequent requests hitting the cache can lower the input token price and reduce latency; but caching cannot shorten the total context length, and long histories still occupy the context window and raise latency.

Prompt cache validity typically ranges from only a few minutes to several hours, and discount rates and cache windows may be adjusted at any time.

Token caching (Prompt Caching) is a cost-optimization mechanism in large model APIs; when the same prefix appears repeatedly across multiple requests, the computation result can be cached and charged at a lower rate.

The caching mechanism is technically commonly referred to as KV Cache or Prompt Caching

The general principle for achieving cache hits is to keep the prefix of the request context as stable as possible, placing fixed content such as system prompts and tool definitions at the very front, with per-turn user input appended at the end.

Prompt caching saves pre-computed KV matrices on the server side, reducing the processing of repeated prefixes from O(n²) to O(1) cache reads, enabling up to 90% compression in latency and cost.

Common context pruning strategies include sliding window (retaining the most recent N messages), summary compression (summarizing old messages into a summary), and importance filtering (prioritizing the retention of tool call results).

Cache entries have a validity period of approximately 5 minutes (default) to 1 hour (extendable), after which they are treated as cache misses.

Prompt caching takes effect only if the cached prefix matches strictly, so the design of the context structure directly determines the hit rate.

The cost reduction stems from a rearchitecting of the Agent's context management mechanism, reducing the number of tokens required per session while increasing the reuse rate of prompt caching, rather than an adjustment of billing rates.

Prompt caching is an inference optimization mechanism introduced by mainstream large-model API providers: when the prefix portion of a request is exactly identical to that of a previous request, the server can directly reuse the already-computed KV cache without re-running the forward computation.

Prompt caching is isolated per model rather than per session; measured in real coding sessions, when switching models mid-session, in 82% of cases cache reads fell below 10% of the previous value, and a warm 107000-token prefix became 110000 tokens of uncached input at roughly ten times the cached price.

When contextual retrieval uses prompt caching, the cost is about $1.02 per million document tokens, one-time.

The hard prerequisite for prompt caching to take effect is prefix determinism: any dynamic content located in the prefix (timestamps, user IDs, randomly ordered tool lists) causes the cache key to change, driving the hit rate to zero.

OpenAI's automatic prompt caching gives a 50% to 75% discount on matching prefixes over 1024 tokens, with no explicit declaration required.

The architectural constraint of prompt caching is: stable content (system instructions, context, examples, etc.) goes first and must be byte-identical each time, while variable content goes at the end.

In multi-turn conversations, each new message extends the history backward, so the cache can only match up to the last stable breakpoint; if an implementation inserts a rolling summary, current-turn metadata or reformatted history at the front, the entire prefix shifts and the hit rate erodes.