Sparse Autoencoder

Sparse autoencoders (SAEs) are one of the main technical approaches attempting to disentangle superposition and extract monosemantic features.

The idea behind sparse autoencoders (SAE) is to project the activation values of a given model layer into a higher-dimensional sparse space, and by constraining only a small number of features to be activated at a time, force the model to learn more disentangled, monosemantic features.

Sparse autoencoders (SAEs) introduce sparsity regularization in addition to reconstruction loss, encoding activation vectors as overcomplete sparse representations.

Individual neurons are often polysemantic, i.e. they respond to multiple unrelated concepts at the same time, so subsequent research turned to methods such as sparse autoencoders (SAEs) to extract purer feature directions.

Sparse autoencoders (SAEs) decompose model activations into an overcomplete dictionary of sparse features via a reconstruction loss, but this is only a post-hoc approximation of an already-trained model, and the reconstruction objective is inconsistent with the downstream task objective.

In 2024, Anthropic used sparse autoencoders to extract millions of independent "features" from its Claude 3 Sonnet model, of which about four-fifths, upon inspection, were not closely tracked by any single neuron.

Current mainstream interpretability approaches include post-hoc attribution methods (such as attention visualization, SHAP) and mechanistic interpretability (such as sparse autoencoders, circuit analysis), with the latter attempting to locate identifiable "feature circuits" at the level of neuron activations.

ConSDL is not proposed to completely reject SAEs, but rather to point out the limitations of the linear encoder as a design choice.

Dictionaries trained by SAEs differ significantly across random seeds, undermining the reliability and reproducibility of analyses.

Standard SAEs use a single-pass linear encoder to map activations to sparse coefficients, and the coefficients obtained from a single forward pass are suboptimal relative to per-sample optimization.

The dictionary optimization problem of SAEs is non-convex and has many local optima, so different random seeds cause dictionary atoms to rotate or recombine in concept space.

On real benchmarks in four vision and language settings, ConSDL improves dictionary consistency by an average of 17.6% and coefficient consistency by 35.9% over six SAE baseline methods.

Under the new paradigm, the sparsity constraint of SAEs is regarded as a kind of human preference bias, rather than the only correct decomposition of the network's true internal structure.

In 2024, Anthropic used sparse autoencoders to extract 34 million features from Claude Sonnet and located the internal circuits corresponding to specific concepts.

The features disentangled by SAEs actually have response widths several times coarser than the original neurons, meaning that SAEs may create a mathematical illusion.

Neel Nanda emphasizes that SAEs remain irreplaceable on the residual stream.

Labs such as Anthropic and OpenAI treat sparse autoencoders (SAEs) as the mainstream approach for disentangling superimposed features in neural networks.

Readout scaling refers to applying a diagonal scaling or linear transformation to the activation vector before feeding the activations into the SAE, thereby changing the relative weight of each dimension in the reconstruction objective.

Sparse factors are recovered best when the training process recovers the space in which the factors were generated.

Holding the readout condition number fixed, changing the generative distribution reverses which training coordinate best recovers the known factors.