Zero-Training Upset: Filling the 256d Null Space with PCA Lifts EmbeddingGemma-2 Audio from Rank 26 to #1

In the audio embedding space, how can a model with only 577M parameters outperform 8B-9B multimodal large models? A recent experiment on Reddit offers a surprising answer: by projecting features from intermediate Conformer layers into the null space of the model's final projection matrix, without any training, EmbeddingGemma-2 jumped from 26th place to 1st on the MAEB audio benchmark.

Key Finding: The Final Layer of Text-Aligned Audio Encoders Filters Out Paralinguistic Information

According to the Reddit post, current LLM-driven Omni embedding models show a stark polarization on the MAEB leaderboard: they score as high as 95-99% on multilingual text-to-audio retrieval tasks, but plummet to 1%-6% on unsupervised audio clustering and only about 20% on speech emotion recognition (CREMA-D).

This huge performance gap stems from the alignment process itself: aligning an audio encoder to a text LLM is essentially a filter on the final layer—retaining lexical content (what is said) while discarding paralinguistic and acoustic features such as timbre, pitch, and emotion. In other words, the better a model is at text alignment, the less it can "hear" the emotion in a voice.

Layer-Wise Probing: Different Depths Encode Different Attributes

The authors probed all 12 Conformer layers of embeddinggemma-2 layer by layer, and the results align with existing speech research (Pasad et al.; Bouziane, Mdhaffar, and Estève)—speaker, prosodic, and semantic attributes peak at different depths of the encoder.

• Layer 2 (mean+std pooling): VoxPopuli gender clustering improves from 0.05% in the final output to 66.6% (layer 3)/over 65% (layer 2), and IEMOCAP gender classification from 66.4% to 83.0%.
• Layer 6 (mean pooling): CREMA-D emotion classification doubles from 19.7% to 39.5%.

These data reveal a key fact: the final output layer is not the most informative representation; intermediate layers retain a large amount of acoustic signal suppressed by the alignment process.

Conformer is a speech encoder architecture that combines convolutional neural networks (CNNs) and Transformers, proposed by Gulati et al. in 2020. Its core idea is to use convolutional layers to capture local acoustic patterns (such as phonemes and prosody) and self-attention layers to model long-range dependencies (such as semantic context). A Conformer block typically consists of four modules: a feed-forward layer, multi-head self-attention, a convolution module, and a second feed-forward layer, with residual connections and layer normalization between modules. In EmbeddingGemma-2, after stacking 12 Conformer layers, shallower layers focus more on low-level acoustic features (such as pitch and timbre), while deeper layers gradually abstract content aligned with text semantics. This is exactly why layer-wise probing can find that information such as gender and emotion peaks in intermediate layers—alignment training mainly acts on the final output, while the acoustic representations in intermediate layers are partially preserved.

The Geometric Principle of the 256d Null Space

Why can new information be injected without training? The key lies in the architectural details of embeddinggemma-2: the text backbone hidden layer size is 512, the terminal projection matrix W has shape 768×512, and there is no activation function afterward.

This yields two geometric properties:

1. 512d text subspace: W linearly maps from 512d to 768d, so all text vectors fall within a 512d subspace.
2. 256d null space: Performing SVD on W yields the remaining 256 orthogonal directions (the null space), which text vectors never use.

This means that the dot product of any audio vector injected into the null space directions with all text vectors is zero—naturally not interfering with text embeddings.

Null space is a fundamental concept in linear algebra: for a matrix ( W \in \mathbb{R}^{768\times 512} ), its null space consists of all vectors ( x \in \mathbb{R}^{768} ) satisfying ( Wx = 0 ). Since the rank of ( W ) is at most 512, by the rank-nullity theorem, the dimension of the null space is at least ( 768 - 512 = 256 ). Vectors in the null space are orthogonal to the row space of ( W ), so any vector ( v ) injected into the null space directions satisfies ( W^T v = 0 ), meaning its dot product with all text vectors projected through ( W ) is zero. This orthogonality guarantees that the injection operation does not change the numerical values of text embeddings, which is the mathematical basis for the "zero-training" method to preserve text invariance.

SVD (singular value decomposition) is the tool for achieving this: performing SVD on ( W ) gives ( W = U\Sigma V^T ), where the columns of ( U ) form an orthogonal basis of ( \mathbb{R}^{768} ). The first 512 ( U ) columns correspond to non-zero singular values and span the text subspace; the remaining 256 ( U ) columns correspond to zero singular values and span the null space. Projecting audio features onto these 256 directions is mathematically equivalent to "bypassing" the filtering mechanism of text alignment.

Zero-Training PCA Injection: Text Unchanged, Audio Tasks Greatly Improved

The authors used PCA to compress layers 2 and 6 to 256 dimensions (fitted on 1400 segments from 4 external audio datasets, without using MAEB/MSEB data), injected them into the null space, and then normalized. The results are as follows:

• Text embeddings: unchanged bit by bit.
• Text-to-audio retrieval (100+ languages on FLEURS, CommonVoice, Google SVQ): changes within 0.05 points.
• Audio-to-audio tasks: gender clustering jumps from 0.05% to 63.1%, artist retrieval from 55.2% to 93.8%, emotion recognition from 19.7% to 40.2%, and environmental sounds from 38.8% to 56.3%.

The overall MAEB macro average rises from 49.3% to 54.4% (+5.1 points), taking first place from 26th in the pure audio track and rising from 16th to 1st on the full MAEB, ahead of 8B-9B multimodal models.

PCA (principal component analysis) is an unsupervised dimensionality reduction method that projects data onto the directions of greatest variance (principal components) through an orthogonal transformation. In this experiment, the authors performed PCA on the pooled features of layers 2 and 6, taking the first 256 principal components to compress high-dimensional audio features to 256 dimensions. Choosing 256 dimensions is no coincidence: it exactly equals the dimension of the null space, allowing the compressed features to be fully mapped onto the orthogonal basis of the null space. The role of PCA is to remove redundant noise, retain the information with the greatest variance in audio features, and ensure that the dimension of the injected vector matches the null space. Fitting PCA used only 1400 segments from external datasets, without touching MAEB/MSEB evaluation data, avoiding data leakage.

Costs and Boundaries: There Are Sacrifices

The authors candidly list two main limitations:

• No Matryoshka support: Since no retraining was done, orthogonality requires all 768 dimensions to be preserved, so the 512/256/128 truncation options are lost.
• Trade-off for spoken intent: The addition of fixed acoustic features slightly reduces the proportion of the 512d semantic component after normalization, hurting audio-to-audio tasks that need to ignore speaker timbre—spoken topic classification (SIBFLEURS) drops from 57.2% to 21.3%, and spoken intent (MInDS14) from 97.6% to 88.3%.

Matryoshka Representation Learning (MRL) is a training strategy that enables embedding vectors to maintain usable performance under multiple dimension truncations (such as 768, 512, 256, 128), similar to Russian nesting dolls shrinking layer by layer. Its principle is to simultaneously optimize sub-vectors of multiple dimensions in the training loss, forcing information to concentrate in the first few dimensions. EmbeddingGemma-2 originally supported Matryoshka truncation, allowing users to choose dimensions based on storage or compute budgets. Because the null-space injection method does not retrain, the injected audio information is spread across all 768 dimensions; once truncated below 512 dimensions, the null-space components are destroyed and orthogonality cannot be maintained, thus losing this flexibility. This is a practical limitation of zero-training methods in engineering deployment.

Insight: Wasted Signals in Overtrained Models

The value of this experiment lies not only in the ranking improvement but also in demonstrating a methodology: overtrained base models may contain a large amount of untapped signal that can be extracted even without training. The authors note that if actual training were restricted to the orthogonal complement of the null space (U_perp), performance could be further improved.

Complete per-task results and details can be found in the Hugging Face model card (BrunoHays/embeddinggemma-2-nullspace-audio). For AI researchers and engineers, this suggests a low-cost approach to improving embedding models—first check whether your model is wasting dimensions.

One-line Summary

Zero-training injection of intermediate-layer audio features into the projection matrix's null space propels a small model to the top of an audio benchmark.

Paragraph Summary

This article introduces an experiment from Reddit: by compressing audio features from EmbeddingGemma-2's intermediate Conformer layers with PCA and projecting them into the 256-dimensional null space of the final projection matrix, without any training, the model jumps from 26th to 1st on the MAEB audio benchmark, surpassing 8B-9B multimodal models. The key insight is that the text alignment process filters out paralinguistic information such as emotion and timbre, while intermediate layers retain these acoustic features. The geometric properties of the null space guarantee that injected vectors are orthogonal to all text vectors, so text embeddings remain bit-for-bit unchanged and text-to-audio retrieval performance is unaffected, while tasks such as audio clustering, artist retrieval, and emotion recognition improve substantially. The cost is the loss of Matryoshka truncation capability and degraded performance on some tasks that rely on ignoring speaker timbre. The method reveals that overtrained models contain a large amount of untapped signal.