MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
The theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
Timeline (last 90 days)
For an MoE model that selects 8 experts per token, verifying 4 tokens can theoretically touch up to 32 different routed experts.
When evaluating the speedup of MoE speculative decoding, both the number of accepted tokens and the number of unique experts should be reported, to avoid the optimistic bias of a single metric.
The number of unique experts should be counted per MoE layer rather than as a global average, because routing distributions can differ greatly across layers and a global average would mask extreme union cases.
In NVIDIA's Ling-3.0-flash model, each token selects 8 of 512 routed experts in each MoE layer, plus one shared expert.
Lightweight statistics can be inserted into the routing module to record the set of expert IDs selected by each MoE layer within the current batch, with very little overhead.
The end-to-end speedup of MoE speculative decoding depends on two factors: the acceptance rate of draft tokens and the degree of expert overlap within the draft window.
The length and sampling strategy of the draft window need to be traded off against expert overlap; a longer window is not necessarily more worthwhile.
The memory footprint of an MoE model is determined by its total parameter count; the more experts there are, the larger the total weights, and the more pronounced the memory pressure.
When running a 35B-level MoE model locally, weight storage in FP16 requires roughly tens of GB.
Strix mainly does three things: tiered caching of MoE experts by frequency (high-frequency experts in VRAM, all experts in memory), a static routing table placed on the SSD, plus sparse attention and speculative decoding layered on top.
40 more timeline events
All Facts (20)
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
90%VerifiedFlash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedMuse Spark Open uses an MoE architecture, with 816B total parameters and 42B activated parameters.
85%VerifiedQwen3.8-Flash-Next adopts a Mixture of Experts (MoE) architecture with 125B total parameters and 6B activated parameters.
85%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%VerifiedDeepSeek V4 Flash is a cost-effective inference model released by DeepSeek, optimized for speed and cost.
80%VerifiedThe MoE architecture uses multiple expert networks and a routing mechanism to selectively activate only a subset of experts during each forward pass, reducing computational overhead while maintaining large model capacity.
80%