Unverified50% confidenceOpinionExact time
The author considers 12–30 tokens/s to be the comfort zone, with above 30 tokens/s being good; an M5 Max with 48–64 GB can run at approximately 20 tokens/s.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
9/10/2026
First Seen
Valid until: 12/9/2026
Sources
Related Entities
Related Claims
VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.71% similarUnverifiedA 200K token context request requires approximately 25× the KV-Cache memory compared to an 8K token request.69% similarUnverifiedIn practical engineering, a chunk_size between 256 and 1024 tokens and a chunk_overlap between 20 and 100 tokens are fairly common empirical ranges.69% similarUnverified1 token is approximately equal to 0.75 English words, and modern models have extended the context window limit to 128K tokens.67% similarUnverifiedTaking Llama-3 70B as an example, the prefill stage for processing 32K tokens on an A100 80GB GPU takes approximately 800–1200ms, while processing 2K tokens takes only about 50ms.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/887806API
curl https://kongchang.com/api/v1/knowledge/claims/887806MCP
get_claim(id=887806)