Megatron-LM MoE Training Handbook: A Practical Guide, Setting by Setting

Hugging Face engineer Nouamane has released The Ultimate MoE Training Handbook, a hands-on guide for researchers who already have experience training dense models and want to move up to Mixture-of-Experts (MoE) models. The handbook draws on roughly 800,000 GH200 GPU hours of MoE training experience on the Alps cluster, and all charts, scripts, and recommended configurations have been open-sourced.

Structure and Core Content

The handbook has four chapters, with Chapters 2–4 being the main focus:

• Chapter 2: The Megatron-LM Runbook — Covers hardware testing, building the image, training a dense baseline, and then training your first MoE on a single node and comparing it against the baseline. This is the most practical section, essentially a checklist for "getting MoE running from scratch."
• Chapter 3: Deconstructing the MoE Layer — Analyzes the router, routing algorithms, load balancing, experts, block layout, training process, and system implementation one by one. Each component comes with Megatron-LM code, monitoring metrics, recommended configurations, and the choices made by frontier MoE models.
• Chapter 4: Serving and RL — Brings MoE models into inference serving and reinforcement learning pipelines.

The author emphasizes that Chapters 3 and 4 can be read without a GPU, which is friendly to readers who want to understand the principles before getting hands-on.

Why It's Worth Your Attention

MoE has become one of the mainstream architectures for scaling large models, but public end-to-end training guides are scarce. What makes this handbook unique:

1. Based on real large-scale training: Practical lessons from roughly 800,000 GH200 GPU hours — not armchair theory.
2. Setting by setting: It doesn't just tell you what to use; it explains why and which metrics to watch.
3. Transparent methodology: Results from the author's own experiments are clearly labeled; claims about frontier models cite their papers, configurations, or code; toy simulations are also labeled separately.
4. Reproducible resources: 55 charts are published as a gallery, and all scripts and recommended parameters are consolidated into a single Megatron-LM config block.

Who Should Read It

If you've already trained dense models and are about to try MoE, or you're curious about how routing, load balancing, and expert parallelism are actually configured in MoE training, this handbook works well as a desk reference. For readers with no distributed training experience at all, the hardware and image sections in Chapter 2 may require some foundational background first.

The author, Nouamane, has previously contributed to the Ultra-Scale and Smol Training Playbooks, continuing a track record in training-handbook content. The handbook, chart gallery, scripts, and recommended configurations are all public and can be accessed directly on Hugging Face Spaces.

Background

Megatron-LM is NVIDIA's open-source distributed training framework for large models. It supports tensor parallelism, pipeline parallelism, data parallelism, expert parallelism, and other parallel strategies, making it one of the mainstream tools for training MoE models in the industry. Its core design goal is efficient scaling on thousand-GPU clusters, which means it has a large number of configuration options and a steep learning curve. The handbook's choice of Megatron-LM as the practical framework means readers need a basic understanding of its parallel group partitioning, checkpoint format, and launch scripts in order to smoothly transition from a dense baseline to MoE.

The core component of the MoE layer is the router, which computes affinity scores for each expert for every token and selects the top-k experts accordingly. Common routing algorithms include Top-k gating, expert choice, and hash-based static routing. The hard part of routing is load balancing: if tokens concentrate on a few experts, it creates computational hotspots and wastes memory, so auxiliary load-balancing losses or a capacity factor are usually introduced as constraints. Expert parallelism distributes different experts across different GPUs and uses All-to-All communication to dispatch and gather tokens, which is key to implementing MoE training systems.