Random Matrix Theory
A theoretical framework in mathematical physics and probability theory that studies the spectral properties of large random matrices. In machine learning theory, it is applied to derive precise asymptotic analyses of the generalization error of neural networks.
Timeline (last 90 days)
The spiked Wigner model is a classic framework in random matrix theory: the observation matrix can be written as Y = λ x x^T + W, where x is the low-rank signal vector to be estimated and W is a real symmetric Wigner noise matrix (upper-triangular entries independent and identically distributed, with mean zero and variance 1/n), and λ controls the signal-to-noise ratio.
The research introduces the high-temperature Marchenko-Pastur (HTMP) distribution assumption to characterize heavy-tailed properties.
The Marchenko-Pastur law describes that when the elements of an m×n matrix are independently and identically distributed, the eigenvalue density of its sample covariance matrix converges to a deterministic distribution in the high-dimensional limit.
This study is one of the first attempts to systematically apply random matrix theory to provide a rigorous statistical foundation for empirical metrics of DNN weight spectra.
The paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
Deterministic equivalents are a core tool of random matrix theory: when the dimension p and sample size n grow to infinity proportionally (p/n → γ), random quantities converge in probability to deterministic expressions.
Marčenko and Pastur established the asymptotic theory for the eigenvalue distribution of sample covariance matrices in 1967, while random matrix theory was originally developed by Eugene Wigner in the 1950s.
The Marčenko-Pastur law describes the distribution of noise eigenvalues from the perspective of random matrix theory
All Facts (8)
The spiked Wigner model is a classic framework in random matrix theory: the observation matrix can be written as Y = λ x x^T + W, where x is the low-rank signal vector to be estimated and W is a real symmetric Wigner noise matrix (upper-triangular entries independent and identically distributed, with mean zero and variance 1/n), and λ controls the signal-to-noise ratio.
50%UnverifiedThe research introduces the high-temperature Marchenko-Pastur (HTMP) distribution assumption to characterize heavy-tailed properties.
50%UnverifiedThe Marchenko-Pastur law describes that when the elements of an m×n matrix are independently and identically distributed, the eigenvalue density of its sample covariance matrix converges to a deterministic distribution in the high-dimensional limit.
50%UnverifiedThis study is one of the first attempts to systematically apply random matrix theory to provide a rigorous statistical foundation for empirical metrics of DNN weight spectra.
50%UnverifiedThe paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
50%UnverifiedDeterministic equivalents are a core tool of random matrix theory: when the dimension p and sample size n grow to infinity proportionally (p/n → γ), random quantities converge in probability to deterministic expressions.
50%UnverifiedMarčenko and Pastur established the asymptotic theory for the eigenvalue distribution of sample covariance matrices in 1967, while random matrix theory was originally developed by Eugene Wigner in the 1950s.
50%UnverifiedThe Marčenko-Pastur law describes the distribution of noise eigenvalues from the perspective of random matrix theory
50%