Unverified50% confidenceBenchmarkExact time
Concurrent requests amplify the demand for cache slots, causing more frequent LRU evictions; on the T4, four concurrent requests achieved total throughput of 37.0 tokens/s, lower than the 43.2 tokens/s of a single request, with the hit rate dropping from 95.7% to 92.3%.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/18/2026
First Seen
Valid until: 11/1/2026
Sources
Related Entities
Related Claims
UnverifiedOn the T4, --fit plus a 10% cache reached 23.6 tokens/s with a 75.6% hit rate; --cpu-moe alone was 9.6 tokens/s; --cpu-moe plus a 50 MiB cache was only 3.8 tokens/s.71% similarUnverifiedStrata's multi-user concurrency uses a parallel slot mechanism, and the slots do not use speculative decoding, causing single-user speed to drop from 144 to 38 tokens/s70% similarUnverifiedWhen 32 requests share the same 4k-token system prompt, Superfluid's prefix cache hit rate reaches 97%, on par with Ollama, while mlx_lm.server is 0%.67% similarUnverifiedPagedAttention causes KV cache memory waste to occur only in the last unfilled block of a request, and vLLM reports waste of less than 4%.67% similarUnverifiedWith a 256K prefix already cached and 4 users, 95% of requests get their first token within 1.66 seconds.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1060513API
curl https://kongchang.com/api/v1/knowledge/claims/1060513MCP
get_claim(id=1060513)