Unverified50% confidenceFactExact time
Open-weight models can achieve throughput of 420 to 2100 tokens/second on Groq LPUs and Cerebras wafer-scale clusters.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/19/2026
First Seen
Valid until: 1/17/2027
Sources
Related Entities
Related Claims
UnverifiedFor a 32-layer, 4096-dimensional model, the KV Cache for processing 2048 tokens occupies approximately 1 GB of VRAM71% similarUnverifiedUnder this configuration, the 35B model achieves a measured single-user peak speed of approximately 80 tokens/second, with concurrent throughput exceeding 500 tokens/second.69% similarUnverifiedSGLang using official NVFP4 weights with D-Spark speculative decoding achieved an average of approximately 173 tokens/second on Blackwell GPUs, approaching 200 for short tasks, while loading a 262k context window.68% similarVerifiedTaking the LLaMA-2 70B model as an example, a sequence of 2048 tokens requires approximately 4 GB of VRAM for the KV cache alone.68% similarUnverifiedMost models have a single output limit of between 4,000 and 8,000 tokens, with some high-end models supporting up to 32,000 tokens.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1072609API
curl https://kongchang.com/api/v1/knowledge/claims/1072609MCP
get_claim(id=1072609)