Residual Stream

Refusal behavior is largely mediated by a single direction in the residual stream, typically computed as the mean difference vector between harmful and harmless prompts.

The residual stream is a fixed-dimensional vector stream that runs through all layers in a Transformer, in which the model's intermediate computation states accumulate and which can serve as an entry point for reading internal states.

Neel Nanda emphasizes that SAEs remain irreplaceable on the residual stream.

The basic operation of activation steering is to find an activation direction vector representing a certain concept and add it to intermediate-layer activations during inference, thereby intervening in the output without changing the weights.

The residual stream is the core information channel of the Transformer architecture, where attention heads and MLP modules at each layer interact with the residual stream in a read-process-write-back manner.

Within effective component mechanisms, the steering effect is further concentrated in approximately 50% of the residual stream's dimensions, which retain 85%–98% of the component-mechanism baseline effect.

Injecting vectors into the residual stream is a commonly used causal intervention technique in mechanistic interpretability, acting directly on the model's intermediate representations rather than at the vocabulary level.

In language model-driven war games, the model's language output simultaneously determines what an agent attempts to do and what becomes the simulated reality.

The framework treats the residual stream as a communication bus between the model's layers, from which attention heads and MLPs read information and write back.

Arditi-style editing originates from research by Andy Arditi et al. in 2024, whose core finding is that the refusal behavior of large language models can be localized to a single directional vector in the residual stream.