AKV
A serving engine designed for cache engineering that internally manages stateful KV cache reuse, eviction, and restoration, while exposing a stateless interface externally to remain compatible with existing inference serving APIs.
Timeline (last 90 days)
Currently AKV's public information is concentrated at the method level, and the specific interface design, scheduling strategy and adaptation results on different models remain to be tested by the community.
Official data shows that AKV reduces average cost by 27.7% across 4 agent benchmarks.
AKV extends SGLang with cache editing capabilities, allowing agents to actively discard or relocate cache segments.
The 27.7% average cost reduction comes from the average across 4 agent benchmarks, and gains may differ noticeably across task types.
A research team at Tsinghua University has open-sourced AKV, an LLM agent cache engineering service system built on SGLang.
AKV allows agents to discard and relocate KV cache during inference rather than relying on summarization to compress history.
On the AKV engine, output throughput improves by up to 1.2x and P95 request latency speeds up by 1.5x.
The open-source code of the AKV engine has been released on GitHub, providing a reproducible foundation for subsequent research.
The paper's authors designed the AKV serving engine, which internally manages stateful cache reuse, eviction, and restoration while exposing a stateless interface externally that is compatible with existing serving APIs.
All Facts (9)
Currently AKV's public information is concentrated at the method level, and the specific interface design, scheduling strategy and adaptation results on different models remain to be tested by the community.
50%UnverifiedAKV allows agents to discard and relocate KV cache during inference rather than relying on summarization to compress history.
50%UnverifiedOfficial data shows that AKV reduces average cost by 27.7% across 4 agent benchmarks.
50%UnverifiedAKV extends SGLang with cache editing capabilities, allowing agents to actively discard or relocate cache segments.
50%UnverifiedThe 27.7% average cost reduction comes from the average across 4 agent benchmarks, and gains may differ noticeably across task types.
50%UnverifiedOn the AKV engine, output throughput improves by up to 1.2x and P95 request latency speeds up by 1.5x.
50%UnverifiedThe paper's authors designed the AKV serving engine, which internally manages stateful cache reuse, eviction, and restoration while exposing a stateless interface externally that is compatible with existing serving APIs.
50%UnverifiedA research team at Tsinghua University has open-sourced AKV, an LLM agent cache engineering service system built on SGLang.
50%UnverifiedThe open-source code of the AKV engine has been released on GitHub, providing a reproducible foundation for subsequent research.
50%