Grokking
Grokking is a counterintuitive training dynamic in deep learning in which a model's test-set performance remains near chance level for an extended period after memorizing the training set, then suddenly improves to strong generalization after many additional training steps. The phenomenon was first widely observed on modular arithmetic tasks.
Timeline (last 90 days)
The new study finds through systematic experiments that the decline in the effective rank of weight matrices follows rather than causes grokking.
The coefficient of determination R² for this power-law fit is 0.732, which improves to 0.821 after adding interaction terms.
The study reveals that when weight decay λ ≳ 1.0, the system crosses from a region where grokking does not occur to a region where it does, forming a sharp phase boundary.
The study observes that weight norms exhibit a monotonically compressing trajectory during the grokking transition.
Weight decay can be viewed as a knob controlling the grokking phase transition; by adjusting λ, it is theoretically possible to actively trigger or suppress grokking.
Grokking refers to the phenomenon where, after a model has already achieved near-perfect fit on the training set, test set performance remains at a random level for a long time before generalization ability suddenly improves after a large number of additional training steps.
The grokking phenomenon was initially widely observed on modular arithmetic tasks.
The study fitted a power-law scaling relationship for the generalization onset time as T_grok ∝ H^(-0.27) D^(-2.04) η^(-0.50) λ^(-0.64), where H is model width, D is data complexity, η is learning rate, and λ is weight decay.
The exponent for data complexity (-2.04) is far larger than that for model capacity (-0.27), meaning that doubling the data can accelerate generalization by approximately 4×, whereas doubling model width yields only about 1.2× speedup.
The dominant factor determining the timing of the grokking transition is data complexity rather than model capacity.
3 more timeline events
All Facts (13)
The new study finds through systematic experiments that the decline in the effective rank of weight matrices follows rather than causes grokking.
50%UnverifiedThe coefficient of determination R² for this power-law fit is 0.732, which improves to 0.821 after adding interaction terms.
50%UnverifiedThe study reveals that when weight decay λ ≳ 1.0, the system crosses from a region where grokking does not occur to a region where it does, forming a sharp phase boundary.
50%UnverifiedThe study observes that weight norms exhibit a monotonically compressing trajectory during the grokking transition.
50%UnverifiedWeight decay can be viewed as a knob controlling the grokking phase transition; by adjusting λ, it is theoretically possible to actively trigger or suppress grokking.
50%UnverifiedGrokking refers to the phenomenon where, after a model has already achieved near-perfect fit on the training set, test set performance remains at a random level for a long time before generalization ability suddenly improves after a large number of additional training steps.
50%UnverifiedThe grokking phenomenon was initially widely observed on modular arithmetic tasks.
50%UnverifiedThe study fitted a power-law scaling relationship for the generalization onset time as T_grok ∝ H^(-0.27) D^(-2.04) η^(-0.50) λ^(-0.64), where H is model width, D is data complexity, η is learning rate, and λ is weight decay.
50%UnverifiedThe exponent for data complexity (-2.04) is far larger than that for model capacity (-0.27), meaning that doubling the data can accelerate generalization by approximately 4×, whereas doubling model width yields only about 1.2× speedup.
50%UnverifiedThe dominant factor determining the timing of the grokking transition is data complexity rather than model capacity.
50%UnverifiedMost mature system design resources on the market (such as the System Design Interview series and Grokking) focus primarily on traditional backend systems.
50%UnverifiedEstablished system design resources on the market, such as the System Design Interview series and Grokking, mostly focus on traditional backend systems and provide insufficient coverage of the unique challenges specific to AI/ML.
50%UnverifiedClassic system design resources such as the System Design Interview series and Grokking mostly focus on traditional backend systems and provide insufficient coverage of AI/ML-specific challenges.
50%