[KongchangAI]
· 2 min read· 1,082 words

Delayed Effects of Temporary Weight Decay in Muon Pretraining

Delayed Effects of Temporary Weight Decay in Muon Pretraining

Temporary weight decay in Muon shows delayed effects: loss rises at removal but ends lower.

A Muon optimizer pretraining study reveals that temporary weight decay interventions have delayed effects: validation loss may rise when the decay window is removed, yet ends lower at the training budget's conclusion. Validated on 162M–3.4B parameter Transformers, a schedule transferred from 162M to 1.4B ultimately beats the no-decay baseline. Mechanistically, early decay causes a persistent weight norm deficit that enlarges relative step size and alters convergence; norm reset experiments confirm the deficit explains most final improvement.

Introduction

In large language model pretraining, the choice of optimizer and hyperparameter scheduling often determines final performance. A recent study on the Muon optimizer reveals a counterintuitive phenomenon: temporary weight decay interventions may increase validation loss when removed, yet actually lower loss at the end of the training budget. This delayed effect challenges the conventional practice of evaluating optimization strategies based on performance at the removal point. This article analyzes the study's core findings, mechanistic explanations, and implications for pretraining practice.

Muon pretraining study schematic

Core Findings: Delayed Effects and Scale Consistency

The research team conducted experiments on decoder-only Transformers ranging from 162M to 3.4B parameters. For models up to 1.4B parameters, shifting a fixed-duration weight decay window from early to late training reverses its final effect—early decay causes loss to rise at removal but results in lower final loss; late decay shows the opposite. More strikingly, a decay schedule selected on the 162M model, transferred without adjustment directly to the 1.4B model, was 0.26 nats worse than the no-decay baseline after removal, only overtaking at approximately 73% of training completion, yet finishing 0.028 nats better than the baseline—while also surpassing the best constant decay baseline.

These results indicate that the effects of temporary optimization interventions do not manifest immediately upon removal, but may fully materialize only in later training or even at completion.

Weight Decay is a regularization technique in deep learning optimization that prevents overfitting by proportionally shrinking weights at each parameter update. In standard implementations, weight decay is typically coupled with the optimizer's update step, for example applied independently of gradient updates in AdamW. Temporary weight decay refers to enabling decay only within a window during training, then disabling it. This intervention temporarily reduces the weight norm, but after disabling, the weight norm takes time to recover, producing sustained effects. Understanding this helps grasp the physical meaning of "delayed effects" in this article: the end of the decay window is not the end of the influence, but the beginning of weight norm dynamics adjustment.

Mechanistic Analysis: Weight Norm Deficit and Relative Step Size Dynamics

Researchers attribute the delayed effect to a persistent weight norm deficit left by early decay. After the decay window ends, the weight norm remains below the no-decay path, causing the relative step size (i.e., the ratio of update magnitude to weight) to remain large in subsequent training, thereby affecting the convergence trajectory. A recursive model approximating norm dynamics can capture the slow convergence process of relative step size. Through fixed-norm replay experiments, the performance ordering and similar loss differences were reproduced, validating the critical role of norm dynamics.

Norm reset experiments on 162M and 405M models further show that the smaller weight norm at the end of the decay window explains most of the final loss improvement. Additionally, similar Gram interventions also exhibit post-removal norm and relative step size dynamics, suggesting this mechanism may be universal.

Relative Step Size refers to the ratio of the norm of the optimizer's single-step update to the current weight norm, i.e., (|\Delta w| / |w|). It measures the relative magnitude of parameter updates rather than absolute size. In adaptive optimizers, relative step size directly affects training dynamics: a larger relative step size means parameters move more aggressively in weight space, potentially accelerating early convergence but increasing risk of late-stage oscillation. Weight decay increases relative step size by shrinking the denominator (|w|). Therefore, the weight norm deficit caused by early decay continuously amplifies relative step size in subsequent training, altering the convergence trajectory. This concept is key to understanding the delayed effect mechanism and explains why norm reset experiments can reproduce performance differences.

Implications for Pretraining Practice

This study conveys a clear message: when evaluating temporary optimization schedules, assessment must be conducted under the expected final training budget, not merely at the removal point. Many scheduling strategies may perform poorly at removal yet harbor long-term benefits; conversely, improvements at removal may come at the cost of final performance. For adaptive optimizers like Muon, dynamic management of weight norms is particularly critical, and schedule design must account for delayed effects.

Future work could explore how to leverage delayed effects to design more efficient training plans, for example by guiding optimization paths through periodic norm interventions. Meanwhile, the reliability of cross-scale transfer still requires careful validation—although transfer from 162M to 1.4B is ultimately effective, the significant mid-training disadvantage demands sufficiently extended training budgets in practical applications.

Muon Optimizer is an adaptive optimization method for matrix parameters whose core idea is to approximately orthogonalize gradients, making update steps more uniform in parameter space. Unlike element-wise adaptive optimizers like AdamW, Muon considers the overall structure of parameters when updating matrix parameters, typically achieving gradient orthogonalization through Newton-Schulz iteration or similar methods. This design makes Muon more sensitive to changes in weight norms, as its update magnitude is directly related to the ratio with weight norm. The relative step size dynamics mentioned in the article are particularly pronounced under Muon, which is why weight norm management is especially important for Muon schedule design. Understanding this characteristic of Muon helps grasp why temporary weight decay produces pronounced delayed effects under Muon.

Conclusion

Temporary weight decay in Muon pretraining demonstrates the delayed effects of optimization scheduling, rooted in relative step size dynamics triggered by weight norm deficits. This finding not only deepens understanding of optimizer behavior but also calls on the research community to evaluate interventions over longer time scales. For practitioners, this means patience and a global perspective are crucial in pretraining tuning.

One-line Summary

Temporary weight decay in the Muon optimizer exhibits delayed effects: loss rises at removal but is lower at training completion.

Paragraph Summary

Based on a pretraining study on the Muon optimizer, this article reveals the delayed effects of temporary weight decay interventions: validation loss may rise when the decay window is removed, but is actually lower at the end of the training budget. This effect is validated on Transformers from 162M to 3.4B parameters, and a schedule transferred from 162M to 1.4B ultimately outperforms the no-decay baseline. Mechanistically, early decay causes persistently low weight norms, increasing relative step size and affecting the convergence trajectory; norm reset experiments confirm that the norm deficit explains most of the final improvement. The article calls for evaluating temporary optimization schedules under the expected final training budget rather than focusing solely on removal-point performance, and highlights the critical importance of weight norm dynamics management for adaptive optimizers like Muon.

#Muon optimizer#weight decay#pretraining#delayed effects#weight norm#relative step size#LLM training
Share:

Related articles