SANATAN Attention: Rebuilding a KV-Cache-Free Transformer with the Imaginary Unit

When Attention Meets Imaginary Numbers: SANATAN's Radical Attempt

A researcher named Bhargav Patel has released an attention mechanism called SANATAN on Reddit, claiming it requires no KV cache whatsoever and that its mathematical foundation is built on the two square roots of the imaginary unit i. His previous paper was criticized for lacking visual explanations, so this time he included a 6-minute video demonstration attempting to intuitively convey the core idea.

This direction is worth watching because the KV cache has long been one of the main bottlenecks in long-context inference. During autoregressive generation, a standard Transformer must store Key and Value vectors for every generated token, with memory usage growing linearly with sequence length. If this requirement could be eliminated at the attention mechanism design level, the cost structure of long-text inference would fundamentally change.

SANATAN's Core Claims

According to the original post, SANATAN's attention computation is reformulated around the properties of the imaginary unit. "Two square roots of −1" refers to the two solutions i and −i in the complex domain—they are mathematically conjugate to each other, and both yield −1 when squared. The author claims this structure allows attention weights to be "reconstructed" in some way rather than "cached," thereby avoiding storage of historical Key/Value pairs.

To be clear: the original post provides no specific mathematical derivation, experimental data, or quantitative comparison with standard attention. The only information currently available includes the mechanism's name, its core claims, and a GitHub repository link. The repository is at github.com/bhargavpatel431997/sanatan-attention, and interested readers can inspect the code implementation directly.

The two square roots of the imaginary unit i: In the complex domain, the equation x² = −1 has two solutions: i and −i. They are conjugate to each other and satisfy i·(−i) = 1. This conjugate structure is commonly used in signal processing to represent orthogonal components (such as IQ modulation), and in attention mechanisms it could potentially be used to encode positional or phase information, allowing attention weights to be computed directly from the current query and position without storing historical Key/Value pairs. However, the original post does not explain how this property is actually used, so readers should treat its mathematical claims with caution.

Why "No KV Cache" Is a Big Deal

To understand what SANATAN is trying to solve, one must first see the cost of the KV cache. In standard multi-head attention, generating a sequence of length N requires caching Key and Value tensors on the order of O(N), and this overhead is amplified by the number of layers and high-dimensional heads. In real deployments, the KV cache often dominates inference memory, directly limiting batch size and context window.

Existing mitigation approaches include MQA (Multi-Query Attention), GQA (Grouped-Query Attention), sliding window attention, and various KV cache compression strategies. These methods essentially trade off "what to cache" and "how much to cache." SANATAN's proposal points to another path: if attention scores can be computed directly from the current state and positional information, historical Key/Value pairs need not be explicitly retained.

This line of thinking is similar to linear attention and state space models (such as Mamba)—they all attempt to replace memory that grows with sequence length with a fixed-size state. But SANATAN chooses to accomplish this using complex-number structures, which is mathematically novel and also means it must answer a key question: can the computational overhead of complex arithmetic truly offset the savings from eliminating the cache?

How MQA and GQA work: Multi-Query Attention (MQA) lets all query heads share a single set of Key/Value heads, compressing the KV cache to 1/H of its original size (H being the number of heads); Grouped-Query Attention (GQA) is a compromise that groups query heads, with each group sharing one set of KV heads. Both reduce cache usage by decreasing the number of KV heads, but they do not change the fundamental fact that the cache grows linearly with sequence length—they only reduce the constant factor.

Linear attention and state space models: Linear attention approximates softmax with kernel functions and rewrites attention computation in recursive form, thereby replacing the KV cache with a fixed-size state matrix; state space models (such as Mamba) achieve similar effects through linear recurrence and selective state updates. What they share is sacrificing the precise token-to-token interactions of standard attention in exchange for O(1) inference memory. If SANATAN truly wants to eliminate the KV cache, it must also confront this trade-off: how to reconstruct historical information without losing expressive power.

The Author's Iterative Approach Is Worth Noting

From the perspective of community interaction, the way this project has progressed is itself somewhat instructive. The author mentioned that his previous paper was criticized because he "couldn't explain the idea visually," so this time he added a video. This practice of supplementing explanatory materials based on feedback is uncommon among independent researchers' projects—many prefer to defend themselves directly rather than fill in the gaps.

However, a visualization video cannot replace technical argumentation. For a proposal claiming to change the fundamental structure of attention mechanisms, what the community cares about most remains: how do hard metrics such as perplexity, downstream task performance, inference speed, and memory usage compare under equivalent parameter counts and training data? The original post provides no such data.

How to Interpret This Research Signal

At a time when attention mechanism innovations are emerging endlessly, SANATAN belongs to the category of early-stage exploration with "high claims, low evidence." Its value lies in proposing a different mathematical perspective—organizing attention computation using structural properties of the complex domain—rather than being a validated engineering solution.

For developers focused on inference optimization, this direction is worth tracking but not worth betting on prematurely. It is advisable to inspect the implementation details in the GitHub repository directly, focusing on: the actual computation graph of complex-valued attention weights, whether gradient propagation is stable, and memory comparisons at realistic sequence lengths. If the author can subsequently supplement controlled experiments against standard attention, the persuasiveness of this work would increase significantly.

Independent researchers' attempts to challenge fundamental architectures deserve respect in themselves, but the technical community's standard of evaluation has always been evidence rather than vision. Whether SANATAN can move from a Reddit post to reproducible benchmark results depends on whether rigorous experimental supplementation follows.

One-line Summary

A researcher proposes the SANATAN attention mechanism based on the imaginary unit, claiming no KV cache is needed, but lacking experimental evidence.

Paragraph Summary

SANATAN is an attention mechanism claiming to completely eliminate the KV cache, with its mathematical foundation built on the two square roots of the imaginary unit i (i and −i). The KV cache is a major memory bottleneck in long-context inference, and existing approaches such as MQA and GQA all make trade-offs in caching strategy, whereas SANATAN attempts to bypass it at the mechanism level. The author, criticized for his previous paper's lack of visual explanation, added a 6-minute video demonstration this time but provided no experimental data or performance comparisons. This direction overlaps conceptually with linear attention and state space models, but whether the computational overhead of complex arithmetic pays off remains to be verified. It is currently an early-stage exploration with high claims and low evidence; tracking the GitHub repository implementation and follow-up controlled experiments is recommended.