Activation Probe
A lightweight linear classifier attached to the hidden states of a specific layer in a large language model, used to monitor the model's internal states in real time. It maps high-dimensional activation vectors to labels and is commonly applied in AI safety monitoring scenarios.
Timeline (last 90 days)
A bag-of-words classifier trained on the same calls performed close to the activation probe, indicating that most of the gain comes from labeled training rather than from the activations themselves.
Locating the relevant neurons usually requires activation probes or attribution analysis, while verifying whether an edit truly establishes causality relies on ablation experiments and controlled experiments.
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
Attention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
Activation probes are a class of lightweight monitoring tools, usually a small classifier or transformer, trained to read the internal activation vectors of one or several layers during large model inference, thereby determining whether the model is currently invoking a certain type of specific knowledge or is in a certain intentional state.
All Facts (5)
A bag-of-words classifier trained on the same calls performed close to the activation probe, indicating that most of the gain comes from labeled training rather than from the activations themselves.
50%UnverifiedLocating the relevant neurons usually requires activation probes or attribution analysis, while verifying whether an edit truly establishes causality relies on ablation experiments and controlled experiments.
50%UnverifiedInterpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
50%UnverifiedAttention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
50%UnverifiedActivation probes are a class of lightweight monitoring tools, usually a small classifier or transformer, trained to read the internal activation vectors of one or several layers during large model inference, thereby determining whether the model is currently invoking a certain type of specific knowledge or is in a certain intentional state.
50%