Unverified50% confidenceBenchmarkExact time
The baseline BF16 model reaches about 37.5 tokens/s at single concurrency, 714 tokens/s aggregate throughput at 32 concurrent requests, and 61ms first-token latency; after switching to FP8 weights, single concurrency rises to 53.4 tokens/s, and peak throughput at 64 concurrency rises from about 800 to 1,140 tokens/s, about 1.42x.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/19/2026
First Seen
Valid until: 1/17/2027
Sources
Related Entities
Related Claims
UnverifiedUnder high power operation, the 35B MoE model reaches 78 to 107 tokens/s, while the dense 27B model only reaches 22 tokens/s.80% similarUnverifiedOn a Korean NPU server (8 cards, 32 chips in total), VKORE raises the single-user response of the Gemma 4 26B model from 14.1 tokens/s to 53.9 tokens/s, about 3.8 times.77% similarUnverifiedOn the T4, --fit plus a 10% cache reached 23.6 tokens/s with a 75.6% hit rate; --cpu-moe alone was 9.6 tokens/s; --cpu-moe plus a 50 MiB cache was only 3.8 tokens/s.75% similarUnverifiedMost models have a single output limit of between 4,000 and 8,000 tokens, with some high-end models supporting up to 32,000 tokens.75% similarUnverifiedOn the 120-billion-parameter GPT-OSS 120B, token generation improves from 90 to 130 tokens/s, and PP improves from approximately 1,300 to approximately 3,200 tokens/s.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1071380API
curl https://kongchang.com/api/v1/knowledge/claims/1071380MCP
get_claim(id=1071380)