Fisher Information Matrix
A central concept in statistics and information geometry that measures the sensitivity of a probability distribution to changes in its parameters. In natural gradient descent it serves as a preconditioning matrix. Computing the full matrix has complexity O(p²).
Timeline (last 90 days)
The FIM analysis revealed that several fast reactions can only constrain their ratios, not individual rates.
Noisy synthetic estimation validated the prediction: the predicted uncertainties of the nine identifiable rate coefficients were within about 30% of the actual estimates.
The effectiveness of EWC relies on the assumption that parameter importance remains stable across tasks; when cross-user drift causes drastic changes in the input distribution, the Fisher estimate may be distorted, and the penalty term instead constrains the model's ability to adapt to new users.
The Fisher-Rao metric is a Riemannian metric on the space of probability distributions that measures the statistical distance between two probability distributions via the Fisher information matrix.
K-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
The Fisher information matrix measures the sensitivity of model parameters to the output distribution; the higher a parameter's Fisher value, the greater the impact of changing it on the output.
The computational complexity of the full Fisher information matrix is O(p^2), where p is the number of parameters, which is infeasible for modern deep networks.
All Facts (7)
The FIM analysis revealed that several fast reactions can only constrain their ratios, not individual rates.
50%UnverifiedNoisy synthetic estimation validated the prediction: the predicted uncertainties of the nine identifiable rate coefficients were within about 30% of the actual estimates.
50%UnverifiedThe effectiveness of EWC relies on the assumption that parameter importance remains stable across tasks; when cross-user drift causes drastic changes in the input distribution, the Fisher estimate may be distorted, and the penalty term instead constrains the model's ability to adapt to new users.
50%UnverifiedThe Fisher-Rao metric is a Riemannian metric on the space of probability distributions that measures the statistical distance between two probability distributions via the Fisher information matrix.
50%UnverifiedK-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
50%UnverifiedThe Fisher information matrix measures the sensitivity of model parameters to the output distribution; the higher a parameter's Fisher value, the greater the impact of changing it on the output.
50%UnverifiedThe computational complexity of the full Fisher information matrix is O(p^2), where p is the number of parameters, which is infeasible for modern deep networks.
50%