Sparse Autoencoders Reveal How Large Language Models Represent Grammatical Number

How large language models internally represent grammatical knowledge has long been a central question in mechanistic interpretability research. A new study uses sparse autoencoders (SAEs) to conduct an in-depth analysis of the Pythia-70M model, revealing how grammatical number (singular vs. plural) is encoded and propagated within the model, and validates the main findings on Gemma-2-2b and Llama-3.1-8b.

SAE: A Key to Opening the Black Box of Language Models

Sparse autoencoders have become a popular tool in recent years for converting LLM hidden state vectors into lists of human-interpretable features. Researchers believe that SAEs have the potential to reveal the model's internal "mental grammar," which holds theoretical value for both the mechanistic explanation of AI behavior and computational research on human language representation.

The Pythia-70M chosen for this study has fewer than 100 million parameters and was trained on a relatively small corpus. Although the reliability of its generated content is limited, the model still exhibits strong grammatical competence and discourse coherence, making it an ideal subject for studying grammatical representations.

Sparse Autoencoder (SAE) is an unsupervised learning model typically composed of an encoder and a decoder, with an overcomplete hidden layer in between. Its training objective is to minimize the error between the input (e.g., an LLM's hidden state vector) and the reconstructed output, while imposing a sparsity constraint (such as L1 regularization) on the hidden layer so that each input is represented by the activation of only a few hidden units. In mechanistic interpretability research, SAEs are applied to decompose the dense, superimposed neuron activations in LLMs into sparse, human-readable feature directions. Each feature direction corresponds to a semantic or grammatical concept, such as "plural noun" or "legal terminology." By analyzing these features, researchers can locate specific knowledge representations within the model and conduct intervention experiments to establish causal relationships. The advantage of SAEs lies in their ability to automatically discover large numbers of interpretable features, overcoming the bottleneck of manual annotation, making them an important tool in current LLM interpretability research.

Discovery of Plural Features and Intervention Experiments

Researchers used SAEs on Pythia-70M to identify features related to grammatical number, including "plural noun," "plural subject," and "generate plural verb." These features can drive the model to output plural verb forms such as "are," "do," and "have" rather than "is," "does," and "has" when presented with prompts like "the men in the park."

More critically, in intervention experiments: removing these features from the hidden state of the plural subject noun token causes the model to generate singular verbs instead; the reverse operation also holds. This directly demonstrates a causal link between these features and grammatical number processing, not merely a correlation.

Intervention experiments are a core method for establishing causal relationships in mechanistic interpretability research. The basic procedure is: first, identify feature directions relevant to a specific task through SAEs or probes; then artificially modify model activations (e.g., ablating, amplifying, or replacing specific features) and observe changes in model output. In this study, researchers removed plural features from the hidden state of plural subject nouns and found that the model switched to generating singular verbs; conversely, amplifying these features promoted plural verb generation. This "ablate-and-verify" paradigm can distinguish correlation from causation, because merely observing the correlation between feature activation and output is insufficient to prove that the feature drives behavior. Intervention experiments provide stronger causal evidence by actively manipulating internal states and have become one of the standard tools in interpretability research.

Asymmetry in Singular-Plural Representations

The study reveals a fascinating phenomenon: the corresponding "singular noun" and "generate singular verb" features are less effective in the same tasks. This asymmetry suggests that Pythia-70M employs a "privative representation" for English grammatical number—that is, plurality is explicitly encoded while singularity may be the default state.

This finding echoes discussions of markedness in linguistics—plurality is typically regarded as the marked form, while singularity is the unmarked default. The model's internal representations appear to replicate this asymmetry.

Privative representation is a concept in linguistics referring to a situation where, of two opposing categories, only one is explicitly marked while the other exists as a default or unmarked state. For example, in the English noun number system, plurality is typically explicitly marked by adding affixes (such as -s), while singularity is often unmarked and treated as the default form. This asymmetry is known in linguistic typology as a markedness difference. In neural networks, privative representation means that the model may explicitly encode only plural features, while singularity is implicitly represented by the absence of plural features. Intervention experiments support this interpretation: removing plural features leads to singular output, while amplifying singular features has limited effect. This suggests that the model may internally replicate linguistic markedness patterns, providing new evidence for cross-disciplinary research between computational linguistics and neurolinguistics.

Cross-Model Validation and Methodological Contributions

To test the generalizability of the conclusions, researchers repeated the experiments on the larger Gemma-2-2b and Llama-3.1-8b models and obtained similar results. This indicates that the privative representation of grammatical number may not be a special case of small models but may exist more broadly across language models of different scales.

The main contribution of this study lies in proposing a new methodology for using SAEs to understand LLM language encoding and in addressing several technical and methodological challenges. For the field of mechanistic interpretability, this provides a complete path from feature discovery to causal intervention and lays the foundation for future research on other grammatical phenomena.

One-line Summary

A new study uses sparse autoencoders to reveal the privative representation of grammatical number within language models, validated through causal interventions.

Paragraph Summary

A new study uses sparse autoencoders (SAEs) to conduct a mechanistic interpretability analysis of the Pythia-70M model, revealing how grammatical number (singular vs. plural) is encoded and propagated within the model. Researchers identified features related to plural nouns, plural subjects, and generating plural verbs, and through intervention experiments demonstrated a causal link between these features and grammatical number processing: removing plural features causes the model to generate singular verbs, and vice versa. Interestingly, singular features are less effective, suggesting the model employs a privative representation where plurality is explicitly encoded and singularity is the default state. This finding echoes markedness theory in linguistics. The study also reproduced the main results on Gemma-2-2b and Llama-3.1-8b, indicating that this representational pattern may be universal. The work proposes a methodology for using SAEs to study LLM language encoding, offering a new path for understanding models' internal grammatical knowledge.