Effective Rank
A metric measuring the effective dimensionality actually used by a matrix, defined based on the entropy of the singular value distribution or the number of directions covered by energy; it is often used as a proxy for representation complexity or degree of compression.
Timeline (last 90 days)
The new study finds through systematic experiments that the decline in the effective rank of weight matrices follows rather than causes grokking.
No matter how late grokking occurs, the steepest decline in effective rank always appears about 900 steps after grokking onset.
Networks that only memorize without generalizing show a much smaller degree of effective-rank compression.
Effective rank is a metric of the "actually used dimensions" of a matrix, typically defined from the singular value distribution: normalizing the singular values into a probability distribution and computing the exponential of its entropy, or counting the number of directions needed to cover a given fraction of energy.
The monitoring effective rank can serve as a metric for detecting whether a model has entered the generalization phase, but one cannot expect to "force" grokking by enforcing low rank.
The first metric is Effective Rank, which measures in a continuous manner how many directions the Jacobian matrix truly acts upon.
All Facts (6)
The new study finds through systematic experiments that the decline in the effective rank of weight matrices follows rather than causes grokking.
50%UnverifiedNo matter how late grokking occurs, the steepest decline in effective rank always appears about 900 steps after grokking onset.
50%UnverifiedNetworks that only memorize without generalizing show a much smaller degree of effective-rank compression.
50%UnverifiedEffective rank is a metric of the "actually used dimensions" of a matrix, typically defined from the singular value distribution: normalizing the singular values into a probability distribution and computing the exponential of its entropy, or counting the number of directions needed to cover a given fraction of energy.
50%UnverifiedThe monitoring effective rank can serve as a metric for detecting whether a model has entered the generalization phase, but one cannot expect to "force" grokking by enforcing low rank.
50%UnverifiedThe first metric is Effective Rank, which measures in a continuous manner how many directions the Jacobian matrix truly acts upon.
50%