Muon
A deep learning optimizer designed by Jordan Juravsky and collaborators specifically for matrix parameters. Its key innovation is orthogonalizing gradient matrices after momentum updates, based on Nesterov momentum and Newton-Schulz iteration. It has demonstrated faster convergence than AdamW in model training efficiency competitions.
Timeline (last 90 days)
MuonGS uses Gram-Schmidt projection to remove the radial component in each Muon update without fixing the weight norm.
The original Muon optimizer can only achieve logarithmic growth in the cumulative effective learning rate during training, which may limit the model's exploration capability.
DP-Adam is a privacy version that embeds the clipping-and-noising step of DP-SGD into the momentum estimation of the Adam optimizer; DP-Muon is a privacy version that uses the Muon optimizer (based on matrix orthogonalization updates) for matrix parameters.
Zero-capacity rigidity shows that under atomless continuous radial distributions, exact scale-extreme optimality can hold only when the gradient is itself already scale-column isometric.
The Muon optimizer replaces the traditional matrix gradient update with an approximate polar factor.
The Isotropic Curvature Model (ICM) explains the effectiveness of Muon from the perspective of curvature: when gradient orthogonalization minimizes a one-step local loss surrogate, Muon's update direction is reasonable.
Under common continuous radial distributions, the mean-curvature interpretation of ICM can hardly provide non-trivial support for Muon's update, and Muon's optimality becomes extremely rare.
Muon's effectiveness may not be simply attributed to gradient orthogonalization under mean curvature, because that explanation is extremely unstable under general radial distributions.
Muon's excellent performance may stem from its worst-case optimization properties rather than curvature matching in the average sense.
Thinking Machines released the Dimuon optimizer, a Muon variant designed specifically for LoRA fine-tuning, and it has been released in Tinker.
4 more timeline events
All Facts (15)
Muon优化器由Kosson等研究者提出，核心改进在于对梯度更新矩阵进行正交化处理
85%UnverifiedMuonGS uses Gram-Schmidt projection to remove the radial component in each Muon update without fixing the weight norm.
50%UnverifiedThe original Muon optimizer can only achieve logarithmic growth in the cumulative effective learning rate during training, which may limit the model's exploration capability.
50%UnverifiedDP-Adam is a privacy version that embeds the clipping-and-noising step of DP-SGD into the momentum estimation of the Adam optimizer; DP-Muon is a privacy version that uses the Muon optimizer (based on matrix orthogonalization updates) for matrix parameters.
50%UnverifiedUnder common continuous radial distributions, the mean-curvature interpretation of ICM can hardly provide non-trivial support for Muon's update, and Muon's optimality becomes extremely rare.
50%UnverifiedMuon's effectiveness may not be simply attributed to gradient orthogonalization under mean curvature, because that explanation is extremely unstable under general radial distributions.
50%UnverifiedThe Muon optimizer replaces the traditional matrix gradient update with an approximate polar factor.
50%UnverifiedThe Isotropic Curvature Model (ICM) explains the effectiveness of Muon from the perspective of curvature: when gradient orthogonalization minimizes a one-step local loss surrogate, Muon's update direction is reasonable.
50%UnverifiedZero-capacity rigidity shows that under atomless continuous radial distributions, exact scale-extreme optimality can hold only when the gradient is itself already scale-column isometric.
50%UnverifiedThe Muon optimizer was designed by Jordan Juravsky et al. for matrix parameters; its key innovation is applying orthogonalization to the gradient matrix after the momentum update.
50%UnverifiedIn the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.
50%UnverifiedThe decoder and lookup table use the AdamW optimizer (learning rate 1×10⁻⁵), while the recurrent weights use the Muon optimizer (learning rate 0.005, momentum 0.95).
50%UnverifiedThe Muon optimizer, proposed by Jeremy Bernstein et al., is centered on the idea of orthogonalizing gradient matrices via Newton-Schulz iteration.
50%UnverifiedMuon's excellent performance may stem from its worst-case optimization properties rather than curvature matching in the average sense.
50%UnverifiedThinking Machines released the Dimuon optimizer, a Muon variant designed specifically for LoRA fine-tuning, and it has been released in Tinker.
50%