Xtier
An open-source plugin built on the official SGLang framework that provides hybrid inference across CPU memory and GPU memory for MoE models. It requires no source code modification and supports an acceleration package mechanism to improve inference speed.
Timeline (last 90 days)
Xtier is an open-source plugin attached to the official SGLang framework that provides hybrid GPU-memory inference capability for MoE models without modifying the source code.
The official SGLang version is mainly oriented toward pure-GPU deployment, while Xtier adds CPU-memory inference support, allowing MoE models to place inactive expert parameters in memory.
The Xtier installation process is to install the official SGLang first, then install the Xtier plugin; at startup the plugin prints detailed logs, completing the stages of startup, copying weights to GPU memory/memory, and loading experts.
On the uploader's device, Xtier takes about three minutes to start.
Xtier exploits the sparsity of MoE by leaving inactive expert parameters in CPU memory and moving only active experts into GPU memory, at the cost that expert switching requires moving data over the PCIe bus, which may become an inference bottleneck.
Under Xtier's acceleration pack mechanism, the baseline speed of official SGLang plus the Xtier plugin is about 65 tokens/s, and with acceleration packs optimized for specific models it can rise to 137 tokens/s.
All Facts (6)
Xtier is an open-source plugin attached to the official SGLang framework that provides hybrid GPU-memory inference capability for MoE models without modifying the source code.
50%UnverifiedThe official SGLang version is mainly oriented toward pure-GPU deployment, while Xtier adds CPU-memory inference support, allowing MoE models to place inactive expert parameters in memory.
50%UnverifiedThe Xtier installation process is to install the official SGLang first, then install the Xtier plugin; at startup the plugin prints detailed logs, completing the stages of startup, copying weights to GPU memory/memory, and loading experts.
50%UnverifiedOn the uploader's device, Xtier takes about three minutes to start.
50%UnverifiedXtier exploits the sparsity of MoE by leaving inactive expert parameters in CPU memory and moving only active experts into GPU memory, at the cost that expert switching requires moving data over the PCIe bus, which may become an inference bottleneck.
50%UnverifiedUnder Xtier's acceleration pack mechanism, the baseline speed of official SGLang plus the Xtier plugin is about 65 tokens/s, and with acceleration packs optimized for specific models it can rise to 137 tokens/s.
50%