Comparing Routing Mechanisms of 10 Open-Source MoE Models: DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss

Why MoE Routing Mechanisms Matter More Than Parameter Counts

Mixture-of-Experts (MoE) architectures have become the mainstream path for scaling large models, yet most discussions stop at "how many total parameters, how many activated." What truly determines a MoE model's performance and training stability is how it assigns each token to experts—that is, the routing mechanism. A configuration-audit-style comparison published by MarkTechPost systematically reviews the routing designs of 10 open-source MoE models, covering total and activated parameters, expert counts, top-k selection, shared experts, gating functions, and load balancing methods. Rather than piling up parameter tables, this article breaks down each model's trade-offs from the perspective of routing design.

For developers and technology decision-makers, understanding these differences has practical significance: routing determines the distribution of inference costs, the difficulty of load balancing during training, and the predictability of behavior during fine-tuning.

Core Dimensions of Routing Mechanisms

Expert Count and top-k: The Direct Source of Sparsity

The basic logic of MoE is that each token activates only a subset of experts. top-k determines how many experts are invoked per forward pass, while the total number of experts determines the model's capacity ceiling. Together they define the sparsity ratio: the gap between total and activated parameters. A small k with many experts yields high sparsity and low per-token computation, but also leaves little margin for error in routing decisions; a large k approaches the stability of dense models at the cost of higher computation.

Models differ noticeably in their choices along these two dimensions, directly reflecting their prioritization of inference efficiency versus model quality.

Routing collapse is a phenomenon worth distinguishing when understanding sparsity design. It refers to the gating network concentrating most tokens on a very small number of experts early in training, leaving the remaining experts with almost no gradients and their parameters unupdated for extended periods. This causes the model's effective capacity to fall far below what the nominal expert count would suggest, manifesting as slow training loss reduction or stalled validation performance. The smaller the top-k and the more experts there are, the lower the probability that any single expert is selected, and the higher the risk of routing collapse—which is why highly sparse designs must be paired with stronger load-balancing constraints.

Shared Experts: Giving Routing a "Fallback Option"

Shared experts are an important evolution in recent MoE design. They refer to a subset of experts that are always activated for all tokens and do not compete in routing. The value lies in providing a stable channel for general knowledge, so the routing network only needs to distinguish specialized needs, reducing the risk of routing collapse. The DeepSeek^Ad series is a representative practitioner of shared-expert design, and several subsequent open-source models have adopted similar approaches.

The proportion of shared experts, and whether they share gating with routed experts, varies across models—this is the part of the comparison that best reveals design philosophy.

Expert parallelism (EP) is a partitioning approach commonly used in distributed training and inference of MoE, closely related to the shared experts and load balancing discussed above. It places different experts on different devices; each token, after routing, is sent to the device holding the target expert for computation, and the results are then aggregated back. The advantage is that the number of experts can scale with the number of devices, breaking through single-GPU memory limits; the cost is the introduction of all-to-all communication, whose volume depends on how dispersed the routing is—the more imbalanced the load, the more some devices' experts are repeatedly invoked while others idle, and overall throughput is dragged down by the slowest device. Thus the choice of expert count, top-k, and load-balancing strategy effectively also determines communication overhead and hardware utilization under expert parallelism.

Gating Functions and Load Balancing

The gating function determines how tokens are scored against experts; common approaches include softmax gating and sigmoid gating. Load balancing methods address the classic MoE training challenge: routing tends to collapse onto a few experts, leaving the rest under-trained. Auxiliary load-balancing loss is the mainstream solution, but models differ in loss coefficients, whether to use an expert capacity factor, and whether to adopt auxiliary-loss-free balancing strategies.

These details are often simplified in public materials; MarkTechPost's comparison verifies against configuration files, making it more credible than secondhand accounts.

The expert capacity factor is a key parameter in load-balancing implementations. In most MoE implementations, each expert has a preset upper limit on the number of tokens it can process in a single batch—its capacity—equal to "the average number of tokens each expert should process × capacity factor." Tokens exceeding capacity are dropped or bypass the expert, ensuring computation is aligned across devices. A small capacity factor yields high computational efficiency but a high drop rate, potentially hurting model quality; a large one reduces drops but increases memory and compute waste. Together with top-k and total expert count, it determines the actual computation graph of a forward pass and is an important source of configuration differences across models.

Routing Design Orientations of Each Model

According to MarkTechPost's configuration audit, models such as DeepSeek, Qwen, Mixtral, Llama 4, and gpt-oss exhibit several typical approaches to routing design. DeepSeek continues the combination of fine-grained experts plus shared experts, emphasizing routing specialization; the Qwen series favors a pragmatic efficiency balance in expert scale and top-k configuration; Mixtral, as an earlier representative of open-source MoE, adopts relatively simple top-2 routing with high design transparency, becoming an important baseline for subsequent research; Llama 4 and gpt-oss tackle expert parallelism and load balancing at larger scales.

It should be noted that the value of this comparison lies in providing a verifiable configuration checklist, not in concluding "whose routing is better." Routing design is tightly coupled with model scale, training data, and inference hardware; comparing a single dimension out of context is of limited meaning.

Practical Reference Value for Developers

When selecting or fine-tuning MoE models, three aspects of routing configuration deserve attention. First, the scale of activated parameters, which directly determines the baseline for inference memory and throughput; second, the combination of top-k and total expert count, which affects batch processing efficiency and latency; third, the load-balancing strategy—if you plan to continue pre-training or perform large-scale fine-tuning, the auxiliary loss settings will significantly affect training stability.

For teams working on model compression or distillation, the presence of shared experts means some capabilities can be transferred more stably—a practical detail that is often overlooked. MarkTechPost's comparison, grounded in configurations, is suitable as a checklist during model selection, but actual deployment results should still be verified through real benchmarks.

FAQ

Are the activated parameters of a MoE model equivalent to the effective parameter count of a dense model? Not exactly. Activated parameters reflect the scale of the computation path per token, but expert switching and load imbalance introduced by routing add overhead; actual effective capacity also depends on expert utilization and training quality.

Are more shared experts always better? No. Too many shared experts weaken the sparsity advantage of MoE and increase constant computation costs; too few loses the role of a stable general-capability channel. A trade-off between sparsity and stability is needed.

Is auxiliary load-balancing loss necessary? Most MoE training relies on it to prevent routing collapse, but some models explore auxiliary-loss-free balancing schemes. Whether it is needed depends on training scale and routing design and cannot be generalized.

One-line Summary

What matters in MoE models is not parameter counts but the routing mechanism that determines sparsity and training stability.

Paragraph Summary

The article points out that discussions of Mixture-of-Experts (MoE) architectures tend to stay at the level of parameter counts and activated parameters, while what truly determines model performance and training stability is the routing mechanism. It breaks down routing design across core dimensions such as expert count and top-k, shared experts, gating functions, and load balancing, explaining how these choices affect sparsity, the distribution of inference costs, and training difficulty. Based on MarkTechPost's configuration audit of 10 open-source MoE models including DeepSeek, Qwen, Mixtral, Llama 4, and gpt-oss, the article outlines each model's typical orientations in fine-grained experts, shared experts, top-k routing, and load balancing, emphasizing that the comparison's value lies in providing a verifiable configuration checklist rather than judging superiority. Finally, it identifies three things developers should focus on during model selection and fine-tuning: activated parameter scale, the combination of top-k and total expert count, and load-balancing strategy.