Activation Probe
A lightweight linear classifier attached to the hidden states of a specific layer in a large language model, used to monitor the model's internal states in real time. It maps high-dimensional activation vectors to labels and is commonly applied in AI safety monitoring scenarios.
Timeline (last 90 days)
一个在相同调用上训练的bag-of-words分类器表现接近激活探针，表明大部分增益来自有标签的训练而非激活值本身。
Locating the relevant neurons usually requires activation probes or attribution analysis, while verifying whether an edit truly establishes causality relies on ablation experiments and controlled experiments.
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
Attention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
Activation probes are a class of lightweight monitoring tools, usually a small classifier or transformer, trained to read the internal activation vectors of one or several layers during large model inference, thereby determining whether the model is currently invoking a certain type of specific knowledge or is in a certain intentional state.
All Facts (5)
一个在相同调用上训练的bag-of-words分类器表现接近激活探针，表明大部分增益来自有标签的训练而非激活值本身。
50%UnverifiedLocating the relevant neurons usually requires activation probes or attribution analysis, while verifying whether an edit truly establishes causality relies on ablation experiments and controlled experiments.
50%UnverifiedInterpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
50%UnverifiedAttention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
50%UnverifiedActivation probes are a class of lightweight monitoring tools, usually a small classifier or transformer, trained to read the internal activation vectors of one or several layers during large model inference, thereby determining whether the model is currently invoking a certain type of specific knowledge or is in a certain intentional state.
50%