RadixAttention

SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.

SGLang's RadixAttention technology can automatically reuse KV Cache across multiple requests, managing shared prefixes through a radix tree structure to reduce redundant computation.

SGLang achieves efficient KV Cache reuse through innovations such as RadixAttention, and can deliver several-fold throughput improvements when handling complex multi-turn dialogue and Agent Loop scenarios.

SGLang is a high-performance large model inference framework jointly developed by academia and industry, with RadixAttention as its core mechanism.

SGLang的RadixAttention对于共享系统提示（system prompt）的场景可带来数倍吞吐量提升

RadixAttention introduces a globally shared KV Cache pool, manages cache lifecycle through reference counting and LRU eviction strategies, and performs longest-prefix matching in a Radix Tree.

SGLang 引入了 RadixAttention 技术，在某些基准测试中推理速度可比 vLLM 等主流推理框架提升 2-5 倍

RadixAttention prefix caching builds the prompts of multiple requests into a radix tree, so identical prefixes compute the KV cache only once and subsequent requests can reuse it directly.

SGLang uses RadixAttention to cache and reuse multi-turn conversations and shared prefixes, and combined with a frontend language for structured generation, further reduces latency and cost in complex invocation scenarios.

SGLang is known for RadixAttention prefix caching and high-throughput scheduling, and provides entry points for plugins and backend extensions.

Mainstream prefix caching implementations include Anthropic's Prompt Caching, OpenAI's automatic caching, and RadixAttention in inference frameworks such as vLLM.

sglang is a high-performance serving framework optimized for large language model inference that supports continuous batching and RadixAttention, offering throughput advantages over Ollama.

RadixAttention can identify prefix segments shared across different requests and directly reuse cached computation results, reducing redundant computation in scenarios with fixed system prompts, multi-turn conversations, or batch document Q&A, thereby improving throughput and reducing time-to-first-token (TTFT).

SGLang's RadixAttention organizes the KV of prompts and generated results into a RadixTree, with common prefixes at the upper levels; paper data shows that the additional overhead is below 0.3% for workloads without prefix reuse.

RadixAttention manages shared prefixes through a Radix Tree structure to reduce redundant computation.

In concurrent request scenarios sharing the same system instructions, RadixAttention can reduce prefill-stage computation by more than 80%.

RadixAttention automatically reuses KV Cache across multiple requests through a radix tree structure, managing shared prefixes to reduce redundant computation and improve inference throughput.

In concurrent request scenarios sharing system instructions, RadixAttention can reduce computation in the prefill stage by more than 80%.

SGLang is developed by LMSYS, focuses on structured output and complex workflows, and introduces the RadixAttention mechanism, which can automatically detect and reuse the common prefix of prompts

SGLang is a high-performance large model inference and serving framework developed by the UC Berkeley LMSYS team, which optimizes KV Cache reuse efficiency through technologies such as RadixAttention.