Muon Optimizer

The Muon optimizer is a gradient optimization algorithm for training deep learning models, combining momentum-based updates with orthogonalization to improve training efficiency and convergence stability in large-scale neural networks. Its core approach applies Nesterov momentum updates to gradients followed by matrix orthogonalization, thereby maintaining superior geometric properties in the parameter update direction. It is applicable to pre-training large models such as Transformers.