Mechanistic Interpretability

A linear probe is a simple linear classifier trained on model activations, used to detect whether specific behaviors or concepts exist inside the model.

Mechanistic interpretability research aims to reverse-engineer the internal computational processes of neural networks, understanding how a model reaches its conclusions rather than merely observing the conclusions themselves.

Mechanistic interpretability attempts to understand what concepts each neuron and each layer's connections encode inside a neural network.

Anthropic's mechanistic interpretability research attempts to locate the storage positions of specific knowledge at the neural network activation level.

Anthropic's sparse autoencoder research has been able to isolate millions of interpretable features from model activations

Interpretability research currently divides into two major directions, "mechanistic interpretability" and "behavioral interpretability", which are complementary but both still immature.

Interpretability in this context refers to efforts to understand the causal relationship between a model's internal computation and its output behavior, and analysis based on neuron directions falls within the scope of mechanistic interpretability.

Mechanistic interpretability attempts to reconstruct the model's computation at the level of internal structures such as neurons, attention heads and activation vectors, for example by locating the specific direction or circuit responsible for refusing harmful requests.

The hidden state steering experiments provide causal evidence that psycholinguistic factors such as commonness and distinctiveness are encoded inside the models, rather than being merely correlational observations.

Hidden state steering is a causal intervention method in mechanistic interpretability research that locates concept directions in hidden representation space and adds or subtracts vectors along that direction to amplify or suppress the strength of a factor's encoding.

Mechanistic interpretability analysis found that for the same math problem, the translated and localized versions often have markedly different internal reasoning trajectories activated in the model, even when final accuracy is close.

The current mainstream "mechanistic interpretability" uses probes, sparse autoencoders, circuit analysis and other means to locate the neurons or subnetworks corresponding to specific capabilities.

Through tools such as sparse autoencoders and circuit analysis, mechanistic interpretability has already identified specific circuits such as 'indirect object identification' and 'polysemous word disambiguation' in small models.

Shevlin appreciates the work on mechanistic interpretability by Neel Nanda's lab inside DeepMind.

Dictionary learning refers to decomposing high-dimensional internal activation vectors of a model into a linear combination of a set of sparse, semantically interpretable atomic features.

The phenomenon of multi-task reuse of attention heads is one of the central unsolved problems in the current field of mechanistic interpretability.

Representative work in mechanistic interpretability includes the complete circuit reconstruction of the indirect object identification (IOI) task in GPT-2.

Mechanistic interpretability is a subfield of explainable AI research aimed at understanding, at the level of internal circuits within neural networks, why a model makes specific judgments; Neel Nanda and Chris Olah are foundational figures in this field.

Mechanistic interpretability research has found identifiable features within models that correspond to real-world concepts, such as Anthropic's research on the Golden Gate Bridge feature in Claude.

Anthropic's research direction of 'mechanistic interpretability' attempts to reverse-engineer model behavior at the circuit level.