Qwen4
A next-generation large language model under development by Alibaba, utilizing a Mixture-of-Experts (MoE) sparse architecture. The target parameter scale is 5 to 10 trillion, positioning it as the next flagship model in the Qwen (Tongyi Qianwen) series.
Core Facts
The benchmark results of Qwen3.8 Flash Next are official Qwen data and have yet to be independently verified by third parties.
The Qwen series introduces the GQA (Grouped Query Attention) mechanism at the architectural level, reducing KV cache memory consumption compared to standard multi-head attention.
Timeline (last 90 days)
The Qwen repository merged an update that raises the default context for models beginning with Qwen4 from 256K to 1M Tokens, covering variants such as Max, Plus, and Coder, and names such as Qwen4 Max and Qwen4.5 Max appear in the test code.
At the Qwen Amsterdam event, the Qwen4 series was previewed, showing four models: Max, Flash, Plus, and 27B; Qwen research scientist Guo Dongxu revealed that the release timing hinted at by the CTO is next week.
The main model of the architecture used by Qwen4 has 125B parameters, activates only 6B per token, uses a MoE architecture, contains 512 experts, and has a native context of 262K.
The Qwen4 architecture expands a single residual connection into four branches, and in actual testing one of them spontaneously becomes a long-range pathway.
The training cost of the Qwen4 architecture is only one-ninth that of the previous generation.
The Qwen4 architecture switches the optimizer to Moong, and Batch Size Warmup is no longer necessary, which instead saves 18.8% of steps.
Searching Hugging Face for the official Qwen4 repository currently returns 0 results.
Qwen4 is still in training and has not yet been released.
The embedding layer of the Qwen4 architecture uses table lookups based on the current word plus the preceding few words, adding 51 billion parameters while adding almost no compute, and the table can be placed in host memory without consuming VRAM.
According to a leak from the Reddit community, the next-generation model Qwen4 from Alibaba's Tongyi Qianwen team will have a 1M (one million) token context window.
20 more timeline events
All Facts (20)
The benchmark results of Qwen3.8 Flash Next are official Qwen data and have yet to be independently verified by third parties.
65%VerifiedThe Qwen series introduces the GQA (Grouped Query Attention) mechanism at the architectural level, reducing KV cache memory consumption compared to standard multi-head attention.
65%UnverifiedQwen 3.5 Plus comes from Alibaba.
60%UnverifiedThe Qwen repository merged an update that raises the default context for models beginning with Qwen4 from 256K to 1M Tokens, covering variants such as Max, Plus, and Coder, and names such as Qwen4 Max and Qwen4.5 Max appear in the test code.
50%UnverifiedThe training cost of the Qwen4 architecture is only one-ninth that of the previous generation.
50%UnverifiedThe Qwen4 architecture switches the optimizer to Moong, and Batch Size Warmup is no longer necessary, which instead saves 18.8% of steps.
50%UnverifiedThe embedding layer of the Qwen4 architecture uses table lookups based on the current word plus the preceding few words, adding 51 billion parameters while adding almost no compute, and the table can be placed in host memory without consuming VRAM.
50%UnverifiedThe Qwen4 architecture expands a single residual connection into four branches, and in actual testing one of them spontaneously becomes a long-range pathway.
50%UnverifiedQwen4 is still in training and has not yet been released.
50%UnverifiedSearching Hugging Face for the official Qwen4 repository currently returns 0 results.
50%UnverifiedThe main model of the architecture used by Qwen4 has 125B parameters, activates only 6B per token, uses a MoE architecture, contains 512 experts, and has a native context of 262K.
50%UnverifiedThe leak came from a GitHub PR (QwenLM/qwen-code#15762) that contains related configuration information and is accompanied by a leaked screenshot as supporting evidence.
50%UnverifiedThe GitHub PR itself is a change to the qwen-code repository and may involve context configuration for the coding model, but the community interpreted it as a clue to the overall specifications of Qwen4.
50%UnverifiedAccording to a leak from the Reddit community, the next-generation model Qwen4 from Alibaba's Tongyi Qianwen team will have a 1M (one million) token context window.
50%UnverifiedAt present, the appearance of Qwen4 and Kimi3.1 on Arena is only a community observation, and there has been no official confirmation.
50%UnverifiedSome users hoped the next-generation model would "push N-grams to the limit" and expected Qwen4 Flash Next to have a larger N-gram table or more experts.
50%UnverifiedIn the discussion, users expressed anticipation for "the next 27B model," hoping for extensive training in design aspects.
50%UnverifiedAccording to feedback from multiple users in the Reddit community, the Qwen4 family already carries a great deal of expectation before release, and Flash Next, as an "advance guard" version, has successfully paved the emotional road for subsequent products.
50%UnverifiedThe Qwen4 architecture adopts a sparse MoE approach, using large parameter counts to store knowledge while using low activation volume to control inference costs.
50%UnverifiedQwen4, built on a next-generation architecture, has already entered training.
50%