Unverified50% confidenceBenchmarkExact time
On the T4 with --cpu-moe plus a 10000 MiB cache, the hit rate was 95.73% for small batches of up to 8 tokens and 59.91% for batches of 9 to 32 tokens.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/18/2026
First Seen
Valid until: 11/1/2026
Sources
Related Entities
Related Claims
UnverifiedThe author reproduced it on a T4 with the same model and testing method: at 8 concurrency, a single request decodes at 122–124 tokens/s, and the whole server produces 701–708 tokens/s, likewise about a 5.7-fold difference.72% similarUnverifiedTokenRouter's Monitors cap the input of each judgment at about 4k tokens.66% similarUnverifiedUnder conditions of 2944 input tokens and 1024 output tokens, normal mode reaches 29.9 tokens per second, which improves to 48.4 tokens/s after enabling MTP2.66% similarUnverifiedIn the scenario where 16 requests each carry a 3,072-token long prompt, Gemma 4 E2B maintains 1049 tokens/second, while Granite 3.3 2B drops to 732 (0.70x), SmolLM3-3B drops to 725 (0.69x), Phi-4-mini drops to 404 (0.39x), and Qwen3-4B reaches only 373 (0.36x).66% similarUnverifiedIn speculative sampling, when the acceptance rate of candidate tokens is high, the effective number of output tokens per forward pass can reach 2–4, thereby improving the overall generation speed by 1.5x–3x.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1060512API
curl https://kongchang.com/api/v1/knowledge/claims/1060512MCP
get_claim(id=1060512)