Subspace Crystallization
The phenomenon during grokking in which a network weight matrix concentrates key information into a low-dimensional subspace spanned by a small number of top singular directions.
Timeline (last 90 days)
The study proposes the concept of "subspace crystallization", defined as the space spanned by the minimal number of top singular directions in a weight matrix needed to retain 99% test accuracy.
In the Transformer, the number of subspace directions drops from more than a quarter of the available directions to about one tenth during the phase transition.
All Facts (2)
The study proposes the concept of "subspace crystallization", defined as the space spanned by the minimal number of top singular directions in a weight matrix needed to retain 99% test accuracy.
50%UnverifiedIn the Transformer, the number of subspace directions drops from more than a quarter of the available directions to about one tenth during the phase transition.
50%