Unverified60% confidenceFactExact time
Official estimates suggest a 24GB GPU can run approximately 100–140 tokens/second.
2
Sources
60%
Confidence
Medium-term (~90 days)
Relevance
10/5/2026
First Seen
Valid until: 1/3/2027
Sources
Related Entities
Related Claims
UnverifiedIf bandwidth reached 2.4 TB/s, the theoretical speed ceiling for a 100 GB model could reach 24 tokens/s, a 60 GB model 40 tokens/s, and a 35 GB model close to 69 tokens/s.78% similarUnverifiedThe 128K character output limit is approximately equivalent to 30,000–40,000 tokens.78% similarUnverifiedA speed of 4–5 tokens/second is too slow for practical use on an 8GB GPU; a 16GB GPU can improve this to 12–14 tokens/second, entering a practically usable range.76% similarUnverifiedTaking a 70B BF16 model as an example, generating one token requires reading approximately 140 GB of weights, and when fully utilizing the H100's 3.35 TB/s memory bandwidth, the per-token latency is at least 42 milliseconds, with a theoretical TPS upper limit of approximately 24 tokens/second.74% similarUnverifiedWith an approximately 1.6M ultra-long context and 128 concurrent requests, throughput can reach approximately 3,700 tokens/s (aggregated).74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/978630API
curl https://kongchang.com/api/v1/knowledge/claims/978630MCP
get_claim(id=978630)