MuonH
A Muon-based optimizer variant that controls the weight norm through hard constraints, guaranteeing linear cumulative effective learning rate growth, used for LLM pretraining.
Timeline (last 90 days)
The MuonW and MuonH optimizers can guarantee at least linear growth of the cumulative effective learning rate under given conditions.
Under long training cycles, MuonGS outperforms MuonH on validation perplexity.
On dense models, compared with MuonH, MuonGS is estimated to achieve about a 15% token-equivalent speedup when matching validation loss.
All Facts (3)
The MuonW and MuonH optimizers can guarantee at least linear growth of the cumulative effective learning rate under given conditions.
50%UnverifiedUnder long training cycles, MuonGS outperforms MuonH on validation perplexity.
50%UnverifiedOn dense models, compared with MuonH, MuonGS is estimated to achieve about a 15% token-equivalent speedup when matching validation loss.
50%