Unverified50% confidenceFactExact time
DeepInfer's wide expert parallelism spreads 384 experts across 128 chips, with each chip holding only about 7 experts, reducing memory traffic by nearly 88%, and a single GPU can output more than 26,000 words per second.
1
Sources
50%
Confidence
Long-term
Relevance
10/25/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCompressing the context from 128K to 32K may quadruple the number of concurrent users a single GPU can serve, representing a trade-off between throughput and context length.68% similarUnverifiedFor DeepSeek V4.1 Flash 1017, the output tokens needed to run the full Intelligence Index dropped from 250 million to 210 million, a reduction of about 16%.68% similarUnverifiedTo achieve the speeds described in the article, an enterprise-grade deployment of at least 128 GPUs is required.67% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.67% similarUnverifiedDiffbot's approach moves 4,608 of the 15,360 routed experts, plus lookup tables (Engram tables int4 and embedding), entirely to host memory, with the GPU reading them over PCIe via UVA, reducing the footprint per card to 72.4 GiB.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1095678API
curl https://kongchang.com/api/v1/knowledge/claims/1095678MCP
get_claim(id=1095678)