Colibri
An open-source large model inference engine written in pure C with zero external dependencies. It implements a three-tier storage hierarchy (VRAM / RAM / NVMe) combined with an on-demand expert-loading mechanism, enabling Mixture-of-Experts models with very large parameter counts to run on consumer-grade hardware.
Core Facts
Timeline (last 90 days)
The ceiling on inference speed when using the hard disk as the backend is constrained by disk I/O, and the actual experience in long-text and high-concurrency scenarios is limited.
Colibri is an open-source inference engine that uses the hard disk as extended memory, reading model shards on demand, enabling ordinary computers to run ultra-large-scale models (such as 2.8T parameters).
Colibri adopts an on-demand expert loading approach, loading the corresponding model shards from disk only when needed, a strategy that is especially well-suited to the MoE (Mixture of Experts) architecture.
Colibri supports model parameter scales ranging from 744B to 2.8T.
Colibri supports nine major model families, including popular open-source models from both domestic and international sources such as GLM, Kimi, and DeepSeek.
Colibri reached third place on the open-source weekly rankings and has accumulated approximately 37,000 stars.
Colibri officially provides pre-compiled versions for three platforms; users can enter a conversational state by downloading the engine and model and then executing three commands.
Colibri's official stance makes no promises regarding inference speed, but provides a hard guarantee on semantics (output correctness and consistency with the model's semantics).
Colibri's backend supports CUDA, Metal, Vulkan, and NUMA all sharing a single engine.
Colibri provides unified access via coli chat, coli serve, and coli web, supporting models including GLM 5.2/5.3, Kimi K3, DeepSeek V4 Flash, and Qwen3.
13 more timeline events
All Facts (20)
Open-source models publicly release their complete weights, allowing anyone to download, modify, run locally, or even develop commercially.
65%UnverifiedAlibaba's Qianwen is open-sourced under the Apache 2.0 license.
60%UnverifiedThe ceiling on inference speed when using the hard disk as the backend is constrained by disk I/O, and the actual experience in long-text and high-concurrency scenarios is limited.
50%UnverifiedColibri is an open-source inference engine that uses the hard disk as extended memory, reading model shards on demand, enabling ordinary computers to run ultra-large-scale models (such as 2.8T parameters).
50%UnverifiedColibri adopts an on-demand expert loading approach, loading the corresponding model shards from disk only when needed, a strategy that is especially well-suited to the MoE (Mixture of Experts) architecture.
50%UnverifiedColibri supports model parameter scales ranging from 744B to 2.8T.
50%UnverifiedColibri supports nine major model families, including popular open-source models from both domestic and international sources such as GLM, Kimi, and DeepSeek.
50%UnverifiedColibri officially provides pre-compiled versions for three platforms; users can enter a conversational state by downloading the engine and model and then executing three commands.
50%UnverifiedColibri's official stance makes no promises regarding inference speed, but provides a hard guarantee on semantics (output correctness and consistency with the model's semantics).
50%UnverifiedColibri's backend supports CUDA, Metal, Vulkan, and NUMA all sharing a single engine.
50%UnverifiedColibri provides unified access via coli chat, coli serve, and coli web, supporting models including GLM 5.2/5.3, Kimi K3, DeepSeek V4 Flash, and Qwen3.
50%UnverifiedOn 6 RTX 5090 GPUs, Colibri's measured speed reaches 9.0–9.2 tokens/s, while on a 25GB development machine streaming from NVMe the speed is 0.05–0.1 tokens/s.
50%UnverifiedWeight placement of MoE models only determines speed and does not alter precision or routing semantics.
50%UnverifiedColibri's one-layer-ahead prefetching (routing lookahead) achieves a measured prediction hit rate of 71.6%.
50%UnverifiedColibri supports dual-disk acceleration with bandwidth-weighted traffic splitting, yielding an overall speed improvement of approximately 33%.
50%UnverifiedColibri includes a coli web visualization dashboard featuring an Atlas brain view that visualizes 19,456 experts, color-coded by storage tier and with brightness indicating routing heat.
50%UnverifiedColibri makes the following commitments: the default policy will not silently alter progress or routing, speed will only be reduced rather than accuracy when high-speed memory is insufficient, and speculative decoding must be proven worthwhile or it will be disabled.
50%UnverifiedColibri is an open-source large model inference engine written in pure C with zero dependencies.
50%UnverifiedColibri covers 9 model families with parameters ranging from 744B to 2.8T, claiming that frontier models can run starting from 16GB of memory.
50%UnverifiedColibri adopts a three-tier storage hierarchy design of VRAM, RAM, and disk, placing hot experts in VRAM, dense backbone layers and hot experts in RAM, and cold experts on NVMe disk for on-demand loading.
50%