Implicit Regularization
Implicit regularization refers to the phenomenon whereby optimization algorithms such as gradient descent inherently bias toward low-complexity solutions without any explicit regularization term being added. This property is considered one of the important sources of generalization ability in neural networks.
Timeline (last 90 days)
Although over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.
Gradient descent naturally tends to select flatter, simpler solutions from among the many solutions that perfectly fit the training set, and this implicit regularization effect is closely related to generalization ability.
All Facts (2)
Although over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.
50%UnverifiedGradient descent naturally tends to select flatter, simpler solutions from among the many solutions that perfectly fit the training set, and this implicit regularization effect is closely related to generalization ability.
50%