Hybrid HiSparse
A long-context inference optimization solution proposed by the vLLM community, based on Sparse Multi-head Latent Attention (Sparse MLA) and hierarchical memory scheduling. Cold KV cache pages are offloaded to host memory, substantially increasing concurrency without additional hardware.
Timeline (last 90 days)
The vLLM community has introduced the Hybrid HiSparse solution, built on top of Sparse MLA, enabling requests to continue decoding without being forcibly interrupted even after the KV Cache can no longer fully reside in HBM
HiSparse combines sparse indexing to retain only the KV portions that are actually queried, which is fundamentally different from traditional full-offload KV offloading
The solution was validated on the GLM 5.3 model, a single 8×H200 node, and a full 1-million-token context configuration
Under the same host memory with a concurrency setting of 32, traditional KV offloading can only sustain 5–6 requests, whereas Hybrid HiSparse can simultaneously maintain 19–25 requests, improving effective concurrency by approximately 3 to 4 times
The hot pages of Hybrid HiSparse come from ordinary KV blocks in the same memory pool, managed uniformly by the Hybrid Memory Allocator
Hybrid HiSparse resolves resident, hot, and missing data rows in a single pass through a single fused kernel, and that kernel is CUDA-graph capturable
Hybrid HiSparse maintains compatibility with vLLM's existing features including prefix caching, OffloadingConnector, P/D disaggregation import, and MTP
Hybrid HiSparse is planned to be officially released in vLLM v0.30
All Facts (7)
The vLLM community has introduced the Hybrid HiSparse solution, built on top of Sparse MLA, enabling requests to continue decoding without being forcibly interrupted even after the KV Cache can no longer fully reside in HBM
50%UnverifiedHiSparse combines sparse indexing to retain only the KV portions that are actually queried, which is fundamentally different from traditional full-offload KV offloading
50%UnverifiedThe solution was validated on the GLM 5.3 model, a single 8×H200 node, and a full 1-million-token context configuration
50%UnverifiedUnder the same host memory with a concurrency setting of 32, traditional KV offloading can only sustain 5–6 requests, whereas Hybrid HiSparse can simultaneously maintain 19–25 requests, improving effective concurrency by approximately 3 to 4 times
50%UnverifiedThe hot pages of Hybrid HiSparse come from ordinary KV blocks in the same memory pool, managed uniformly by the Hybrid Memory Allocator
50%UnverifiedHybrid HiSparse resolves resident, hot, and missing data rows in a single pass through a single fused kernel, and that kernel is CUDA-graph capturable
50%UnverifiedHybrid HiSparse maintains compatibility with vLLM's existing features including prefix caching, OffloadingConnector, P/D disaggregation import, and MTP
50%