Unverified50% confidenceFactExact time
When the prefix cache is hit, repetitive prefill of tens of thousands of tokens can be almost entirely skipped, dropping first-token latency from seconds to milliseconds.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedUnder the same experimental setup, the total time to generate 1016 tokens drops from 14.66 seconds to 5.03 seconds.79% similarUnverifiedWith 8 concurrent users, each gets 19 tokens/s, for a total of 130 tokens/s, with first-token latency of 2 to 8 seconds76% similarUnverifiedThe time required for a million-token output may reach the level of several minutes, and integration requires designing asynchronous calls and progress feedback mechanisms.76% similarUnverifiedIn a test scenario where eight agents each process 7.5k-token documents and a chat request is issued in the second second, Superfluid answers in 0.36 seconds while llama-server outputs no first token for more than 10 seconds, a first-token latency gap of about 30 times.74% similarUnverifiedIt is generally considered that single-concurrency output above 100 tokens/second is sufficient to achieve a smooth typewriter effect, and at approximately 200 tokens/second, generating a 2,000-character long text takes only about 10 seconds.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1062229API
curl https://kongchang.com/api/v1/knowledge/claims/1062229MCP
get_claim(id=1062229)