Feature Attribution
A family of interpretability methods that measure the contribution of each input dimension to the output, including gradient-based methods.
Timeline (last 90 days)
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
Attention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
The distance between two samples in this representation space directly reflects the magnitude of their difference at the level of feature attribution.
All Facts (3)
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
50%UnverifiedAttention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
50%UnverifiedThe distance between two samples in this representation space directly reflects the magnitude of their difference at the level of feature attribution.
50%