Memorization Effect
The phenomenon whereby over-parameterized neural networks can memorize random labels. However, the implicit regularization of gradient descent biases them toward first learning simple, consistent regularities and only afterward memorizing exceptional cases.
Timeline (last 90 days)
The gain from the sampler can be decomposed into a memory effect (present on both Sudoku and GSM8K) and a token-selection effect (present on GSM8K and most pronounced at small budgets).
The memory effect refers to the sampler's tendency, when re-noising the state, to retain information about already generated tokens, thereby reducing repeated errors.
Although over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.
All Facts (3)
The gain from the sampler can be decomposed into a memory effect (present on both Sudoku and GSM8K) and a token-selection effect (present on GSM8K and most pronounced at small budgets).
50%UnverifiedThe memory effect refers to the sampler's tendency, when re-noising the state, to retain information about already generated tokens, thereby reducing repeated errors.
50%UnverifiedAlthough over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.
50%