Running a 68GB Large Model on a 12GB GPU: Betelgeuse Engine Benchmarked at 22 tok/s

Running large models on consumer hardware is often bottlenecked by VRAM and system memory. A developer on Reddit shared an inference engine called Betelgeuse, claiming it can sustain 22 tok/s on an ordinary gaming PC with an RTX 3060 12GB GPU and 16GB DDR4 RAM while running a 68GB Qwen 3.8 Flash Next MoE model, with a guarantee that outputs are 100% bit-identical to the original model.

Background: Compromises of the Old Approach and New Challenges

The author admits that about two months ago he tried to accelerate CPU/GPU offloading by predicting MoE expert activation, but ultimately abandoned that prototype. The reason: he used gate pruning—dropping "cold" experts when routing weights were low. Although the tok/s numbers looked good, on low-bit quantized models this approach quietly degraded output quality and reasoning ability. He was unwilling to ship a "lobotomized" model just for a screenshot-worthy speed number.

When Qwen 3.8 Flash Next was released, he decided to take on the challenge of running the 68GB GSQ-RCO-IQ2_XS build at usable speed on an ordinary budget gaming PC—without dropping a single expert. The test platform deliberately mirrors a typical 16GB-memory PC: RTX 3060 12GB, single-channel DDR4 16GB (~19 GB/s), CachyOS/Arch Linux (strictly confined within a cgroup with MemoryMax=12G), and a consumer PCIe NVMe SSD (~2.1 GB/s).

MoE (Mixture of Experts) architecture is the prerequisite for understanding this article. A traditional dense Transformer activates all parameters for every token generated, whereas an MoE layer contains multiple parallel "expert" feed-forward networks; a router scores each expert based on the hidden state of the current token and selects only the top-k highest-scoring experts to participate in computation. In the model discussed here, for example, each layer has 10 routed experts, so the model's total parameters can far exceed VRAM capacity while a single inference pass touches only a small fraction of them—this is precisely why "a 68GB model running on 28GB of available memory" is theoretically feasible. The cost: unselected experts still need to live somewhere, and once they are swapped out to slow storage, an I/O bottleneck appears.

Quantization and bit-exactness also deserve explanation. Names like GSQ-RCO-IQ2_XS and Q8_K_XL belong to llama.cpp's GGUF quantization formats, where smaller numbers indicate more aggressive weight compression (IQ2 is roughly 2-bit). Quantization itself introduces error, so "100% bit-identical" means: given the same quantized weights and the same sampling configuration, Betelgeuse's output is byte-for-byte identical to the original inference stack—i.e., the acceleration does not additionally alter model behavior. This contrasts with gate pruning, which directly skips experts deemed "cold"; although it saves compute, it changes the computation result, and on low-bit models the error is amplified.

Limitations of Native llama.cpp and Strata

Loading the 68GB MoE into 12GB VRAM + 16GB RAM in standard llama.cpp gives predictable results: mmap causes severe OS page-cache thrashing, producing roughly 1500 major page faults per token, and generation speed is only 1.47 tok/s.

The author also tested Strata. Strata can reach 40+ tok/s when there is 48GB–64GB of memory to mlock 24GB+ of experts, but in a 16GB environment it refuses to load unless --low-ram mmap is forced; once forced, speed drops to 8.11 tok/s, limited by cold SSD reads.

mmap and page-cache thrashing are the key background for this performance comparison. By default, llama.cpp uses memory mapping (mmap) to access the model file as virtual memory, with the OS paging file contents into physical memory on demand. When the model is far larger than RAM, the working set is constantly swapped in and out, and every access to a non-resident page triggers a "major page fault" that must synchronously wait for a disk read. The "roughly 1500 major page faults per token" mentioned above means that every generated token incurs over a thousand blocking disk accesses—1.47 tok/s is the direct manifestation of this overhead.

mlock is a Linux system call that locks specified memory pages in physical RAM, preventing the kernel from swapping them out. On machines with 48–64GB of memory, Strata keeps 24GB+ of expert data resident via mlock, thus avoiding page-cache thrashing; but on a 16GB machine, there is not enough lockable memory, so it can only fall back to mmap mode, and performance collapses immediately. This also explains why "the same algorithm performs vastly differently across memory capacities."

Core Mechanisms of Betelgeuse

Betelgeuse is a fork of llama.cpp (bel-cli / bel-server), and the author is also submitting PRs upstream. It achieves its breakthrough on limited hardware through several innovations:

Zero Gate Pruning, 100% Bit-Exact

All 10 routed experts in each of the 48 layers are computed at full precision. The author verified byte-for-byte against the original stock output using 28,080 routing records and 576 generated tokens, with a difference of 0.000e+00.

Arrival-Relaxed VRAM Slot Pool (~5000 slots)

All 49 dense layers + GDN + QSA + LM head remain in about 3.2GB of VRAM. The remaining VRAM holds about 5000 expert slots, distributed non-uniformly: early layers 0–4 have 240 slots each (compute slack of 0ms), middle layers have 92 slots, and tail layers have 60 slots.

In-Stream GPU Doorbell + Single CUDA Graph Replay

Instead of letting ggml_backend_sched split the graph 48 times per token to ask the CPU which experts are missing (which consumes about 360ms/token of cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits on-device in about 2µs, ringing the mapped host doorbell only when an expert is missing. Per-layer token embeddings are also staged on CUDA0, allowing the entire decoding process to run as a single CUDA Graph replay.

CUDA Graph is an execution optimization provided by NVIDIA: a sequence of GPU kernel launches and memory operations is pre-recorded into a graph, and each subsequent replay requires only a single submission, thereby eliminating a large amount of host-side launch overhead and synchronization waiting. The problem is that in MoE inference, "which experts are in VRAM" changes dynamically; the traditional approach requires the CPU to check layer by layer and split the computation graph as needed, and this host-device round-trip (the article mentions about 360ms/token of cudaStreamSynchronize) completely cancels out the benefits of CUDA Graph. Betelgeuse's approach is to push the hit determination down to a lightweight kernel on the GPU, notifying the host via a "doorbell" only when an expert is actually missing, so that the entire decoding process can be fixed as a single graph replayed repeatedly.

RAM Victim Cache + io_uring Pager

When an expert is evicted from the VRAM pool, it moves into a dedicated host RAM victim cache rather than being discarded. VRAM and RAM remain 100% mutually exclusive, avoiding duplication of the same hot experts in the OS page cache. Physical NVMe reads drop from about 71 MB/token to 12.43 MB/token.

35GB Expert Bloat Without Duplication

Instead of keeping 35GB of dead expert tensors in both Shard 1 .gguf and experts.bin, the expert payload is stripped from Shard 1, so the main GGUF consists of just 3.2GB (Dense.gguf) + 35.3GB (experts.bin, 4KB-aligned single-read blocks) + 26.8GB (PLE.gguf n-gram shards).

Measured Results

On RTX 3060 12GB + 16GB DDR4 with MemoryMax=12G:

• Stock llama.cpp (mmap): 1.47 tok/s cold, 1.63 tok/s warm, peak about 2 tok/s, prompt processing 14–20 tok/s, physical SSD reads 208–312 MiB/tok.
• Strata (--low-ram mmap): 8.11 tok/s cold and warm, peak about 9 tok/s, prompt processing 34.6 tok/s, pool wait about 105ms, output KL divergence 0.022.
• Betelgeuse (bel-server): 16.80 tok/s cold, 22.03 tok/s warm, peak up to 48 tok/s, prompt processing 16.2–19.3 tok/s, physical SSD reads 12.43 MiB/tok, output 100% byte-exact.

When warm, it sustains 21.8–22.03 tok/s on long coding prompts; when generation hits a stretch where all 10 routed experts reside in VRAM/victim pool, speed can spike to 48 tok/s.

Even on the Q8_K_XL quantization of about 190GB total size, bel-server can cold-start at 8 tok/s and warm-start at 12 tok/s (peak 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 machine, whereas stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.

Future Plans

The author is porting this streaming + victim cache architecture to DeepSeek^Ad V4.1 Flash (1017), and early tests have already reached about 25 tok/s on comparable hardware. The project is open-sourced on GitHub (Betelgeuse-AI/Betelgeuse), with a pre-packaged Alpha-Orionis-XS 3.2GB Dense+Experts bundle available on HuggingFace, and a PR has been submitted upstream to llama.cpp.

For developers with an ordinary gaming PC, this may mean being able to run larger, stronger MoE models locally without sacrificing output quality.

One-line Summary

Betelgeuse lets an ordinary PC with a 12GB GPU + 16GB RAM run a 68GB MoE model at 22 tok/s with bit-exact output.

Paragraph Summary

The developer released the inference engine Betelgeuse, which sustains 22 tok/s on an ordinary gaming PC with an RTX 3060 12GB plus 16GB DDR4 while running a 68GB Qwen 3.8 Flash Next MoE model, with peaks up to 48 tok/s and a guarantee of 100% bit-identical output to the original model. His previously abandoned gate pruning approach was fast but harmed low-bit model quality. By comparison, stock llama.cpp achieves only 1.47 tok/s due to mmap page-cache thrashing, and Strata forces mmap under low memory for 8.11 tok/s. Betelgeuse uses zero-pruning full-expert computation, a VRAM slot pool, an in-stream GPU doorbell with single CUDA Graph replay, a RAM victim cache, and io_uring paging to cut physical SSD reads from about 71MB/token to 12.43MB/token. The author is porting it to DeepSeek V4.1 Flash, has open-sourced it, and has submitted a PR upstream.