Sparse Attention
A technical approach for improving the Transformer attention mechanism by retaining only high-value attention connections rather than computing full pairwise token interactions, thereby reducing computational complexity from O(n²). Common implementations include sliding-window attention and hybrid global-plus-local attention, and the technique serves as a core method for enabling ultra-long context processing.
Core Facts
Timeline (last 90 days)
The industry typically mitigates the computation and memory issues of long sequences with techniques such as FlashAttention, sparse attention, sliding windows, chunked processing, and KV cache compression.
To maintain usable efficiency at the tens-of-millions context scale, it is usually necessary to introduce optimizations such as sparse attention, chunked retrieval, state space models or hybrid architectures.
Current mainstream sparse attention implementations can be divided into three categories: full reach, sliding windows, and learned selectors.
GLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
Sparse attention reduces computational complexity to near-linear by having each token attend only to a subset of the sequence.
Adopting sparse attention can compress the KV cache to approximately 1/4, supporting approximately 4 times the context length of full attention under the same memory budget.
Sparse attention may miss critical information for tasks requiring long-range dependencies, so in practice it is often combined with retrieval-augmented generation (RAG).
Many linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
An overly long context dilutes the model's attention to the current task, and trimming the context often leads to more focused and accurate AI output.
Existing mainstream sparse attention methods filter interactions by retaining entries with larger values or high-quality regions in the attention matrix.
4 more timeline events
All Facts (17)
Informer reduces the self-attention computation complexity from O(n²) to O(n log n) through ProbSparse sparse attention.
70%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%UnverifiedThe industry typically mitigates the computation and memory issues of long sequences with techniques such as FlashAttention, sparse attention, sliding windows, chunked processing, and KV cache compression.
50%UnverifiedTo maintain usable efficiency at the tens-of-millions context scale, it is usually necessary to introduce optimizations such as sparse attention, chunked retrieval, state space models or hybrid architectures.
50%UnverifiedCurrent mainstream sparse attention implementations can be divided into three categories: full reach, sliding windows, and learned selectors.
50%UnverifiedGLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
50%UnverifiedAdopting sparse attention can compress the KV cache to approximately 1/4, supporting approximately 4 times the context length of full attention under the same memory budget.
50%UnverifiedSparse attention may miss critical information for tasks requiring long-range dependencies, so in practice it is often combined with retrieval-augmented generation (RAG).
50%UnverifiedSparse attention reduces computational complexity to near-linear by having each token attend only to a subset of the sequence.
50%UnverifiedMany linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
50%UnverifiedAn overly long context dilutes the model's attention to the current task, and trimming the context often leads to more focused and accurate AI output.
50%UnverifiedSparse attention allows each query to interact with only k keys (where k is much smaller than n), reducing complexity to O(nk).
50%UnverifiedExisting mainstream sparse attention methods filter interactions by retaining entries with larger values or high-quality regions in the attention matrix.
50%UnverifiedThe paper reformulates sparse attention as a matrix approximation problem rather than a problem of selecting the largest values from a bag of entries.
50%UnverifiedIndustry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
50%UnverifiedSpeculative decoding improves inference speed by using a small draft model to generate candidate tokens that are verified in parallel by the main model; sparse attention reduces computational complexity from O(n²) to near O(n).
50%Unverified滑动窗口注意力（如Mistral采用的方案）通过限制每个token只关注固定窗口内的历史，将复杂度从O(n²)降至O(n·w)
50%