8GB VRAM Runs Qwen 3.8 Flash Next: 900 t/s Measured

Running large models locally often hits VRAM as the hardest constraint. A Reddit user ran Qwen 3.8 Flash Next on a laptop GPU with 8GB VRAM and 32GB RAM, achieving up to 900+ t/s prompt processing and 30+ t/s generation, and outperforming their previous Qwen 3.6 35B A3B fine-tune in real coding tasks. The core of this setup is Strata—an inference framework that distributes model weights across VRAM, RAM, and SSD.

Hardware and Framework: Fitting a 125B Model into 8GB VRAM

The poster's configuration is not luxurious: RTX 5060 Laptop (8GB VRAM), Ryzen 9 8945HX (16 cores, 32 threads), 32GB DDR5-5200 RAM, and a 1TB SSD with ~5200/4200 MiB/s read/write, PCIe Gen 4 x8, running Linux. Previously, Strata was discussed in the community as typically requiring at least 12GB VRAM, so whether 8GB can replicate similar results is the most direct point of interest in this share.

Strata's approach is not to fit the entire model into VRAM, but to keep resident experts in VRAM, map experts that don't fit to SSD via --mmap-experts, and allocate KV cache between VRAM and RAM. The poster used the Gyro S quantized version of Qwen 3.8 Flash Next, citing it as the best balance between quality and speed—previously tried quantizations like GSQ-RCO Coder and GSQ-RCO IQ2_XS either fell into repetition loops or struggled noticeably with non-English capabilities.

According to the poster, Strata itself does not affect answer quality; it only handles resource allocation and expert residency. What truly determines output quality is the quantization version.

MoE (Mixture of Experts) architecture is the premise for understanding this setup. Qwen 3.8 Flash Next is an MoE model with a total of 125B parameters, but each inference only activates a small subset of "expert" networks. This means VRAM doesn't need to hold all expert weights simultaneously; inactive experts can remain in RAM or on SSD and be swapped in on demand. Strata's --mmap-experts leverages this: it places expert weights on SSD as memory-mapped files, and the GPU only reads the relevant shard when a corresponding expert is hit. MoE's sparse activation makes "running a 125B model on 8GB VRAM" theoretically possible, whereas a dense model would collapse in speed even with the same offloading, because every layer must read all weights.

GGUF and K-quant quantization is another thread. Versions like Gyro S and GSQ-RCO IQ2_XS essentially compress model weights to lower bit widths (e.g., 2-4 bit), trading a small amount of accuracy for a large reduction in size and bandwidth. Lower quantization bit width reduces SSD and RAM read pressure, but quality degradation becomes more obvious—the repetition loops and degraded non-English capabilities the poster encountered are typical side effects of low-bit quantization.

Key Parameters: Tuning Two Values Yields the Most Noticeable Performance Gain

The poster provided the full configuration and emphasized that for their hardware, the two most beneficial parameters are --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB. The former reserves VRAM for other tasks; 500 is the minimum that works, but they recommend 900. The latter reserves RAM for the system, and Windows users should set it higher.

Other notable settings include:

• --kv-resident 20480: lower bound for KV cache allocation; lower values leave more headroom for VRAM.
• --pcie-frac 0.35: controls the proportion of GPU reads over PCIe when each layer misses VRAM; can be calibrated with ./setup.sh --calibrate.
• Do not enable MTP: although it can boost generation speed, it consumes VRAM and lowers prompt processing speed, which is not worthwhile for this configuration; consider it only with 10GB+ VRAM.
• Run Vision on CPU: about 1-2 seconds per image, but saves considerable VRAM, and Strata runs Vision in a separate process that can be swapped out without significantly slowing the model.

On Linux, they also created a separate 32GB swap partition and used systemd's MemorySwapMax=0 to pin the service process in RAM, preventing large contexts and conversation caches from swapping out the service and causing read/write performance jitter.

KV cache is the VRAM hog in long-context inference. When generating each token, the Transformer needs to cache the Key and Value vectors of all previous tokens to avoid recomputation. The longer the context, the larger the KV cache, which can easily reach tens of GB at 262K context. --kv-resident 20480 specifies the lower bound of resident KV cache (in MiB); Strata places the excess in RAM and the GPU reads on demand. This also explains why enabling --parallel with large contexts is risky: multiple concurrent sessions each hold a copy of the KV cache, and both VRAM and RAM can be exhausted simultaneously.

PCIe bandwidth is the hidden bottleneck of expert offloading. The RTX 5060 Laptop uses PCIe Gen 4 x8, with theoretical bandwidth of about 16 GB/s, far below VRAM bandwidth. --pcie-frac controls the proportion of direct GPU reads over PCIe when each layer misses VRAM; a higher ratio may improve hit rate, but PCIe transfer becomes the new bottleneck. ./setup.sh --calibrate measures the actual throughput of the local PCIe and SSD and provides recommended values—this is why copying parameters is risky: differences in PCIe lane counts across motherboards and SSD performance directly change the optimal configuration.

Measured Speed: Still 800+ t/s at 262K Context

The poster provided measured ranges for two configurations. The Chat configuration (single session) at 262K context achieves 800-970 t/s prompt processing and 30-40 t/s generation; at 131K context, 930-1,040 t/s and 30-45 t/s. The Agentic configuration (multi-session and delegation) at 262K context achieves 520-740 t/s and 30-40 t/s; at 131K context, 670-890 t/s and 30-45 t/s.

As a comparison, their previous Qwen 3.6 35B A3B on llama.cpp with VRAM-constrained expert layout and MTP/Vision disabled achieved 630-680 t/s prompt processing and 27-35 t/s generation. In other words, the new setup achieves higher prompt processing speed and slightly better generation speed with lower VRAM usage.

If using local models in multi-session scenarios, the poster warns against enabling --parallel with large contexts (above 131K), as it consumes both VRAM and RAM heavily—at best causing a sharp speed drop, at worst OOM. A safer approach is --conversation-cache: suspend a limited number of sessions and resume them sequentially, avoiding re-reading long contexts when switching sessions. Related parameters include cache size 4096 MiB, 2 slots, minimum cache tokens 8000, and minimum free memory reserved 1024 MiB.

Prompt processing (prefill) and generation (decode) are two phases with different characteristics. Prompt processing is parallel computation: with inputs of thousands to hundreds of thousands of tokens, the GPU can batch process them at once, yielding high throughput, mainly limited by compute and VRAM bandwidth. Generation is token-by-token autoregressive; each step must read all activated weights and KV cache, making it inherently a serial bottleneck, so generation speed is typically only a fraction of prompt processing speed. The gap between 900+ t/s and 30+ t/s in the article reflects this principle. For long-context coding agents, prompt processing speed often affects perceived responsiveness more than generation speed—each new session must re-read a large context, and slow prefill directly drags down interaction rhythm.

Quality Validation: Real-World Performance in Coding Agent Tasks

The poster did not run standardized benchmarks, but spent a week comparing on real tasks, including Baota, Flappy Bird clone, and Pelican tests in the Pi Agent environment, where the Pelican test also compared different reasoning_effort settings. The comparison target was their current Cyber-Tiel 35B A3B (based on Ornith 1.5 Heretic), which often fell into repetition loops, prompting the switch from Tiel to Cyber-Tiel.

The conclusion is that all their coding agents have switched to this model, with noticeable improvements in development speed and code quality. Due to confidentiality, more tests cannot be made public, and they leave broader benchmarking to the community. The poster also raises a question: is there still value in continuing to use Qwen 3.6 35B A3B for agent tasks?

It should be noted that this configuration is highly dependent on specific hardware and system tuning, and copying parameters is risky—improper settings may affect system stability or cause the model to fall into heavy swap. Windows users should lower memory allocation; throughput will drop noticeably, but OOM risk also decreases. If you have more than 8GB VRAM, you can try higher-bit quantizations like Gyro M/L, and treat this configuration as a starting point rather than an endpoint.

FAQ

Can 8GB VRAM really run Qwen 3.8 Flash Next? According to the poster's configuration, yes—provided you use Strata for expert offloading, map some experts to SSD, and choose low-bit quantization like Gyro S.

Is 30 t/s generation speed enough? For local coding and agent tasks, it's in the usable range. Prompt processing speed is the more critical metric in long-context scenarios, and this setup maintains high levels even at 131K context.

Does it work on Windows? Yes, but leave more RAM for the system; throughput will drop, prioritizing stability. All of the poster's tests were done on Linux.

One-line Summary

An 8GB VRAM laptop runs a 125B MoE model with the Strata framework, beating a 35B setup in speed.

Paragraph Summary

A Reddit user ran Qwen 3.8 Flash Next (125B MoE) on an RTX 5060 Laptop (8GB VRAM) and 32GB RAM using the Strata inference framework, distributing expert weights across VRAM, RAM, and SSD. With Gyro S quantization, the Chat configuration achieved 930-1,040 t/s prompt processing and 30-45 t/s generation at 131K context, still 800+ t/s at 262K context, comprehensively surpassing the previous Qwen 3.6 35B A3B setup. The author notes that --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB are the two most beneficial parameters, and recommends using --conversation-cache instead of --parallel for large contexts to avoid OOM. After a week of real coding agent task validation, the author has switched all coding agents to this model, while warning that the configuration is highly hardware-dependent and copying it carries risk.