Linear Attention

Self-attention weights are normalized through the Softmax function, and the sum of the weights is always 1.

自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²

Qwen3 27B combines three linear attention layers with one full attention layer in a recurrent structure at a ratio of 3:1.

Linear attention or SSM layers can process long sequences with near-linear complexity, but they are often weaker than full attention on precise recall and complex reasoning.

Linear attention models (such as Mamba and RWKV) need only maintain a fixed-size recurrent state at inference time rather than caching the entire history of key-value pairs, so memory usage is independent of sequence length.

The recursive state-compression structure of linear attention brings constant memory usage and linear time, but all historical information must be squeezed into a fixed-size memory, and interference easily occurs when new information conflicts with old information.

KronLSTM provides an update mechanism for associative matrix memory by introducing the Kronecker Delta Rule, improving long-context performance while keeping memory dynamics stable.

In linear attention, kernel functions are used to decompose softmax attention into an associable form, reducing inference computational complexity from quadratic in sequence length to linear.

Linear Attention rewrites the attention computation using associative kernel functions or state recurrence, compressing historical information into a fixed-size state matrix, so that each step only needs to update this state, reducing computational complexity from O(n²) to O(n), with constant-level memory usage.

Linear attention-style architectures use a fixed-dimensional recurrent state to accumulate historical information; this state is a lossy compression of the context read so far, with a dimension far smaller than the original KV cache.

Different types of attention mechanisms (such as softmax attention, linear attention, etc.) belong to the same family in the statistical mechanics sense and their phase structure can be analyzed with a unified framework.

GLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。

Linear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.

Many linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.

Gated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.

Minimax has trained a 456B-parameter MoE using a linear attention approach, but still requires occasional use of full attention.

Linear attention, sparse attention, and State Space Models (such as Mamba) can reduce computational complexity to linear levels, opening the door to million-token context windows.

DeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.

Qwen3.8 Flash Next adopts a hybrid architecture whose core technology is Gated DeltaNet, combining linear attention with standard full attention

Hybrid architectures (linear attention + standard attention) are more efficient than pure full attention on long-context tasks, but linear attention may be slightly inferior to standard attention on certain reasoning tasks that require precise global dependencies