How Token Caching Can Save You 98% on Your DeepSeek Bill

Every time you call a large model API, you're paying for "re-reading" the same content. A Bilibili creator broke down DeepSeek's token caching mechanism in a video under two minutes, revealing the startling secret behind wildly different bills: for the same ~100K tokens read, some pay less than half a cent while others pay dozens of times more.

Large models have no memory—they "read from the start" every time

Large models fundamentally have no persistent memory. Every time you send a new message, the system must repackage every previous round of the conversation and send it to the model as input. This means the longer the conversation, the more input tokens each request carries, and costs balloon accordingly.

Input and output are priced completely differently. Input is read all at once as a whole block—fast and cheap per unit; output can only be generated one token at a time—slow and more expensive per unit. The original article notes that output is priced at four times input. So the bulk of the cost in long conversations often isn't in the sentence you just wrote, but in the repeatedly retransmitted historical context.

Tokens and billing units

A token is the basic unit of text a model processes—not strictly a "character" or a "word." The common approach is to split text into subword fragments: a common English word may take 1 token, while rare words get split into several; Chinese typically maps to about 1–2 tokens per character, depending on the tokenizer and text content. Input and output prices on API bills are both billed by token count, not by request count.

Context window

The total number of tokens a model can read in a single pass has an upper limit, called the context window. In multi-turn conversations, the system concatenates historical messages, the system prompt, and the new question into one overall input; once it approaches the limit, history must be truncated or summarized. This is also the root cause of why "repackaging all history every time" keeps driving costs up: the longer the conversation, the more tokens get packed in.

Disk cache: content you've already read isn't recomputed at full price

DeepSeek^Ad's solution is to store read content in a disk cache. When the beginning of a new request matches historical content exactly, that part is read directly from cache—a "cache hit"; only the newly added portion needs recomputation—a "cache miss."

The price gap is orders of magnitude: a cache hit costs only one-fiftieth of a miss, storing cache is free of extra charge, takes effect within seconds, and requires no code changes from developers.

The creator demonstrated with a real bill: one call with about 97,000+ input tokens, the vast majority of which was old conversation already stored, with only the latest 1,800+ tokens actually missing. Looking at the input portion alone, without caching it would cost one mao (0.1 yuan); with caching it's under four li (0.004 yuan)—26 times cheaper.

Prompt caching

This type of mechanism is commonly called prefix caching or prompt caching in the industry: the server saves the input prefix of a request along with its internal computation results (key-value cache and other intermediate states), and if the next request has an identical prefix, it's reused directly, skipping redundant computation. What it mainly saves is compute and time, which is why the price can be significantly lower than a miss.

How hits and misses are determined

The determination is based on token-by-token comparison of the prefix: starting from the first token, only consecutive identical portions count as a hit; once a difference appears, everything after is counted as a miss. So the cache hit rate depends on "how long the common beginning is and whether it's stable," not on whether the overall content is similar.

Two hard constraints of caching

Caching isn't a cure-all. It only recognizes a "character-for-character identical beginning"—change one character at the start, and all subsequent cache is invalidated. This explains why a tiny change to the system prompt can send costs soaring.

The other constraint is "best effort": caching doesn't guarantee a hit every time. Additionally, prices double during weekday peak hours, further amplifying the cost of cache misses.

TTL and cache eviction

Behind "best effort" is cache lifecycle management. Server-side caches typically have a time-to-live (TTL); entries not accessed for a long time get cleaned up. Cache capacity is also limited, so old entries are removed according to an eviction policy. This means even with a completely identical prefix, a miss can occur if too much time has passed or server load is high. In engineering practice, the usual approach is to put stable prefixes first, place variable content later, and call continuously within the cache's validity period to maximize hit probability.

The essence of saving money is "preserving the prefix"

The creator concludes: not a single character less of the content that should be read is read, but 98% of the cost of re-reading is saved. For developers, the optimization direction is clear: keep the conversation prefix stable, avoid frequently changing the system prompt, and place variable content as late as possible.

This also indirectly answers the question in the title—a bill 50 times higher often isn't the model playing favorites, but a world of difference in cache hit rate. Understanding the token lifecycle cuts costs more directly than switching to a cheaper model.