Cumulative Effective Learning Rate
A theoretical metric for measuring an optimizer's exploration capacity, defined as the nominal learning rate divided by the cumulative sum of the weight norm over the course of training, reflecting the total magnitude of relative weight displacement.
Timeline (last 90 days)
The original Muon optimizer can only achieve logarithmic growth in the cumulative effective learning rate during training, which may limit the model's exploration capability.
Theoretical analysis shows that under MuonGS the weight norm grows by at most a factor of √t after t steps, guaranteeing a cumulative effective learning rate C_T = Ω(√T).
The cumulative effective learning rate C_T = Σ_{t<T} η_t / ||W_t||_F links weight-norm control with the optimizer's exploration capability.
All Facts (3)
The original Muon optimizer can only achieve logarithmic growth in the cumulative effective learning rate during training, which may limit the model's exploration capability.
50%UnverifiedTheoretical analysis shows that under MuonGS the weight norm grows by at most a factor of √t after t steps, guaranteeing a cumulative effective learning rate C_T = Ω(√T).
50%UnverifiedThe cumulative effective learning rate C_T = Σ_{t<T} η_t / ||W_t||_F links weight-norm control with the optimizer's exploration capability.
50%