SGLang
SGLang (Structured Generation Language) is an open-source inference framework for large language models, designed to improve both execution efficiency and programming flexibility in LLM serving. Key features include efficient KV cache reuse via RadixAttention, support for structured output constraints such as JSON mode, and a concise front-end programming interface for constructing complex multi-step inference pipelines. SGLang is compatible with multiple mainstream model architectures and GPU hardware, suitable for high-performance inference deployment in both academic research and production environments.
Core Facts
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
SGLang is a high-performance framework designed for large language model inference, developed by researchers from UC Berkeley and other institutions.
Timeline (last 90 days)
In continuous batching frameworks such as vLLM and SGLang, the decoding steps of multiple requests are packed into large batches, and the verification stage of speculative decoding competes for compute with other requests in the batch, causing relative gains to decline.
When running the Llama 3.1 8B model on an i7-100, SGLang reaches about 16200 tokens/s and vLLM about 12500 tokens/s, a difference of about 30%.
vLLM and SGLang are both under the Apache 2.0 license, both provide OpenAI-compatible interfaces, and both support Prefix Cache, speculative decoding, multi-GPU parallelism, and so on.
The source material does not provide a complete technical root-cause analysis of exactly which code path caused this leakage.
AKV extends SGLang with cache editing capabilities, allowing agents to actively discard or relocate cache segments.
A research team at Tsinghua University has open-sourced AKV, an LLM agent cache engineering service system built on SGLang.
SGLang uses RadixAttention to cache and reuse multi-turn conversations and shared prefixes, and combined with a frontend language for structured generation, further reduces latency and cost in complex invocation scenarios.
Both vLLM and SGLang support features such as continuous batching and tensor parallelism, multiplying the number of requests a single GPU can serve concurrently.
Model compression, inference framework optimizations (such as vLLM and SGLang), and the spread of dedicated inference chips have made the compute consumption of the same API call far lower than half a year ago.
DeepInfer will not replace vLLM and SGLang, because it does not do prefix caching or online serving, but it can serve as a reference showing which state must be stored and which can be recomputed.
40 more timeline events
All Facts (20)
Muse Spark Open can be called on the day of release through platforms such as Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI.
90%VerifiedSGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
85%VerifiedSGLang is a high-performance framework designed for large language model inference, developed by researchers from UC Berkeley and other institutions.
80%VerifiedThe inference stack is the software layer between hardware and business applications in large model deployment, covering modules such as model loading and weight sharding, KV Cache management, continuous batching, quantized inference, and request scheduling.
75%VerifiedSGLang (Structured Generation Language) is a high-performance serving framework for large language models and vision-language models.
75%VerifiedSGLang's RadixAttention technology can automatically reuse KV Cache across multiple requests, managing shared prefixes through a radix tree structure to reduce redundant computation.
75%VerifiedSGLang achieves efficient KV Cache reuse through innovations such as RadixAttention, and can deliver several-fold throughput improvements when handling complex multi-turn dialogue and Agent Loop scenarios.
75%VerifiedAn inference framework is a software system dedicated to efficiently running already-trained large models, with core tasks including computation graph optimization, operator fusion, memory management, batching and scheduling, and quantized inference.
70%VerifiedIn an RTX 5070 (12GB VRAM) configuration, the local Qwen3 9B model's output speed reached approximately 70–76 tokens/second.
70%VerifiedThe vLLM team integrated the new kernels open-sourced by DeepSeek into the inference engine, further improving underlying computational efficiency.
65%VerifiedSGLang is a high-performance large model inference framework jointly developed by academia and industry, with RadixAttention as its core mechanism.
65%VerifiedDeepGEMM is a high-performance matrix multiplication library designed by DeepSeek specifically for its MoE models, supporting low-precision formats such as bf16 and fp8.
65%UnverifiedSGLang由斯坦福团队开发，专注于结构化输出和复杂推理链的高效执行
90%UnverifiedSGLang由UC Berkeley LMSYS团队开发，于2024年初开源
85%UnverifiedNIXL是SGLang用于Prefill-Decode分离部署的关键通信组件，负责在Prefill节点和Decode节点之间传输KV Cache状态
85%UnverifiedSGLang由UC Berkeley团队主导开发，是目前与vLLM并列的两大主流开源LLM推理框架之一
85%UnverifiedSGLang通过优化的张量传输协议和零拷贝技术降低KV Cache跨节点迁移延迟
85%UnverifiedSGLang的RadixAttention对于共享系统提示（system prompt）的场景可带来数倍吞吐量提升
75%UnverifiedBy copying CPU weight views to contiguous memory before H2D transfer, full model loading time was reduced from approximately 35 minutes to 6 minutes 20 seconds, achieving a 5.6× speedup.
70%UnverifiedThe SGLang project is maintained by the sgl-project team.
70%