Qwen4 Architecture Preview Open-Sourced: 125B Parameters, 512 Experts, Designed for VRAM-Constrained Hardware

Release Status: Official Confirmation Is Only "Currently Training"

Regarding the release status of "Qianwen 4" (i.e., Qwen4), claims online are extremely muddled. After verifying official information, only one thing can be confirmed: Qwen4 is still in training and has not been released.

According to a summary by a Bilibili creator in a video, the official announcement at the "Yunxi Conference" covered chips, simultaneous interpretation audio, and video models, but Qwen4 only appeared by name, with no specific release date announced. Official materials mentioned model names such as Max, Flash, Plus, and 27B, but did not list specifications like 35B A3B, so these can only be considered "semi-confirmed."

How can you verify this yourself? The creator offered a simple method: search for the official Qwen4 repository on Hugging Face; currently, there are 0 results. This "zero results" is itself the most direct signal — no weights means no real release.

Architecture Preview: What Qwen3.8 Flash Next Reveals

Although Qwen4 itself hasn't arrived, its architecture has been released as an "early preview." Qwen3.8 Flash Next, released on August 26, was explicitly described by the official team as an early preview of the architecture used by Qwen4. For developers focused on local deployment, this config.json is worth a close look.

Core Specs: 125B Total Parameters, Only 6B Activated per Token

Several key metrics can be read directly from config.json:

• Main model: 125B parameters, but only 6B activated per Token
• 512 experts (MoE architecture)
• Native context: 262K
• Training cost is only one-ninth of the previous generation

Behind these numbers is a clear orientation: designed for VRAM-constrained hardware. A 125B model would previously require multi-GPU parallelism, but a 6B activation level means inference requires significantly less VRAM.

MoE (Mixture of Experts) architecture is key to understanding this set of parameters. In traditional dense models, every Token passes through all parameters during inference, whereas MoE splits the feed-forward layer into multiple "expert" sub-networks, with a routing network selecting only a few experts for each Token. Therefore, "total parameters" measures model capacity and knowledge reserves, while "activated parameters" determine the compute and VRAM overhead of a single inference pass. 125B total parameters with 6B activation means the model is large in size, but only about 6B weights actually participate in computation per Token, so VRAM usage and latency are close to those of a 6B-class model. The trade-off is that all expert weights still need to be loaded into memory, which is exactly the problem addressed later by the Ngram table and host-memory approach. With 512 experts, granularity is finer and routing more flexible, but the demands on load balancing and communication scheduling are also higher.

Four Architecture Changes, All Pointing to "Saving VRAM"

The official notes say the architecture changed in four aspects, and the creator interpreted each one:

Attention mechanism: Of the four attention components, three use GDNA, and the remaining one uses QSA for precise retrieval. QSA selects content by micro-block rather than by Token, saving even the index overhead.

Residual connections: A single residual path is expanded into four branches; in testing, one of them spontaneously becomes a long-range pathway.

Embedding layer: The most unconventional design — using the current word plus the preceding few words to look up a table, adding 51 billion parameters while barely increasing compute, and it can be placed in host memory, consuming no VRAM at all.

Optimizer: Switched to Moong, which also revealed that Batch Size Warmup is no longer necessary, saving 18.8% of steps.

All four changes point to one conclusion: this architecture pursues higher inference efficiency under limited VRAM rather than simply stacking parameters.

The attention mechanism is the part of the Transformer where compute and VRAM overhead grow rapidly with context length. Standard full attention at 262K context requires computing associations between each Token and all historical Tokens, and the KV cache also balloons dramatically. The GDNA and QSA mentioned here belong to sparse/selective attention approaches: only some attention heads perform global computation, while the rest focus on a few relevant segments through block-level or retrieval-based selection. QSA's "selecting by micro-block rather than by Token" means indexing is done in units of contiguous small blocks, so the index overhead itself grows with the number of blocks rather than the number of Tokens, thereby reducing both computation and index VRAM at large context. Only when this design is combined with MoE's low activation level does a 262K native context become feasible on consumer-grade VRAM.

Can It Run Now? Yes, but It's the Preview Version

Qwen4 has no weights, so there is no "Qwen4-style" entry point. What can be experienced now is the Qwen3.8 Flash Next preview, which already powers Qwen Studio Chat, Qwen AI, and the Standard mode of Qwen Work.

To get the weights, search for "Qwen3.8 Flash Next" on Hugging Face; downloads have already reached 3.63 million.

Local Deployment Threshold: Runs with 24G of Free Memory

There is an unavoidable threshold for local deployment: the Ngram table is a separate 47.7G file, larger than the 10G that is activated. The community uses host memory as a fallback; measured data is as follows:

• With 64G memory, latency increases by only 6%, the most cost-effective option
• Compressed to 24G, it uses only 6G of VRAM, with latency stable at 9.1 milliseconds

In other words, it can run with just 24G of free memory. However, the license is not Apache 2.0 but the Qwen Community Model License; commercial use requires opening and confirming the terms.

The Ngram embedding table can be understood as a huge lookup table mapping "word sequences to vectors": the model no longer relies solely on the neural network to compute each word's representation on the fly, but instead uses a fragment consisting of the current word and the preceding few words to directly look up pre-trained vectors from the table. The advantage is that table lookup consumes almost no compute, and the table can reside in host memory and be swapped in on demand without occupying VRAM; the cost is that the table itself is extremely large (47.7G here), requiring sufficient memory or high-speed storage. This also explains why, in testing, latency increased by only 6% with 64G of memory and remained at 9.1 milliseconds even when compressed to 24G — the bottleneck has shifted from VRAM to memory bandwidth and paging efficiency.

Three Verification Points and How to Judge

Combining the creator's summary, three actionable criteria can be distilled:

1. Qwen4 is still in training; online claims that it has been "released" are all rumors.
2. The architecture has been open-sourced: 125B, 6B activated, 512 experts; the architecture identifier in config.json is Qwen4.
3. The most unconventional part is the Ngram embedding, which reveals a design for VRAM-constrained hardware.

To judge authenticity, check only three places: whether Hugging Face has a Qwen4 repository, whether Model Studio has Qwen4 Max, and whether Qwen AI has a related blog post. Only when all three appear is it a real release.

If you want to explore the architecture, run Qwen3.8 Flash Next; if you want to wait for the real Qwen4, keep an eye on the 27B tier — the previous generation's 27B just became the most-liked open-source model on Hugging Face, and community expectations for this tier are the highest.