Unverified50% confidenceBenchmarkExact time
Under the same host memory with a concurrency setting of 32, traditional KV offloading can only sustain 5–6 requests, whereas Hybrid HiSparse can simultaneously maintain 19–25 requests, improving effective concurrency by approximately 3 to 4 times
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
9/15/2026
First Seen
Valid until: 12/14/2026
Sources
Related Entities
Related Claims
UnverifiedHiSparse combines sparse indexing to retain only the KV portions that are actually queried, which is fundamentally different from traditional full-offload KV offloading75% similarUnverifiedReducing the KV cache from FP16 to INT8 saves 50% of VRAM, and reducing it to INT4 saves 75%.65% similarUnverifiedA 200K token context request requires approximately 25× the KV-Cache memory compared to an 8K token request.63% similarUnverifiedModel weights cannot fill up VRAM entirely, as space must be reserved for the KV cache and inference processes; the KV cache required for a 128K context can reach as high as several GB or even exceed the model weights themselves.63% similarUnverifiedIn traditional LLM inference, the KV Cache for each request requires pre-allocated contiguous GPU memory blocks; because generation length is unpredictable, memory must be reserved at the maximum length, resulting in actual GPU memory utilization that is typically below 30%.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/921501API
curl https://kongchang.com/api/v1/knowledge/claims/921501MCP
get_claim(id=921501)