FlashAttention
A technique that reduces the memory footprint and computational complexity of the Transformer attention mechanism through IO-aware tiled computation. It is one of the key technologies enabling extremely large context windows.
Core Facts
Flash Attention was proposed by Stanford's Tri Dao et al. in 2022, reducing memory complexity from O(N²) to O(N) through tiled computation and recomputation.
Flash Attention is an attention computation optimization algorithm proposed by Stanford University that processes data in chunks, keeping intermediate results in the GPU's high-speed SRAM to reduce memory bandwidth pressure and lower VRAM usage.
FlashAttention was proposed by Tri Dao et al. and improves attention computation speed by 2–4× through an IO-aware tiled computation strategy.
Timeline (last 90 days)
The industry typically mitigates the computation and memory issues of long sequences with techniques such as FlashAttention, sparse attention, sliding windows, chunked processing, and KV cache compression.
Efficient operators such as FlashAttention have been deeply integrated into mainstream frameworks such as PyTorch, TensorRT-LLM, and vLLM.
FlashAttention mainly reduces memory usage and memory access counts, with the most obvious gains on long sequences, but limited improvement on short sequences or when memory access is not the bottleneck.
Graph generation models often use graph-specific Transformer architectures that jointly update node and edge features, but they struggle to directly leverage hardware-aware attention implementations such as FlashAttention, becoming a scalability bottleneck.
MLX's kernel design approach has been reused on CUDA, potentially forming an optimization path distinct from mainstream implementations such as cuBLAS/FlashAttention.
FlashAttention is an IO-aware exact attention algorithm that fuses the attention matrix computation into SRAM through tiling and online softmax, maintaining numerical exactness while reducing memory usage and improving speed, and has become the default implementation in mainstream inference frameworks.
Token pruning or compression methods that rely on attention weights are difficult to combine with FlashAttention, because FlashAttention does not materialize and store the attention matrix during computation, making it hard for external modules to read directly.
Online Softmax is the key algorithm behind efficient attention implementations such as FlashAttention, avoiding materialization of the full [B, H, S, S] matrix through streaming recurrence.
ModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
The maturation of inference optimization technologies such as vLLM, FlashAttention, and continuous batching has driven a sustained decline in the computational cost per token.
26 more timeline events
All Facts (20)
Flash Attention was proposed by Stanford's Tri Dao et al. in 2022, reducing memory complexity from O(N²) to O(N) through tiled computation and recomputation.
90%VerifiedFlash Attention is an attention computation optimization algorithm proposed by Stanford University that processes data in chunks, keeping intermediate results in the GPU's high-speed SRAM to reduce memory bandwidth pressure and lower VRAM usage.
90%VerifiedFlashAttention was proposed by Tri Dao et al. and improves attention computation speed by 2–4× through an IO-aware tiled computation strategy.
90%VerifiedFlashAttention was proposed by Tri Dao and colleagues at Stanford University; by performing attention computation in tiles within the GPU's on-chip SRAM, it reduces memory access count by an order of magnitude and is a mathematically exact-equivalent implementation.
85%VerifiedThe core innovation of FlashAttention is to reduce the memory access complexity of standard attention computation from O(N²) to near-linear by controlling the data transfer pattern between the GPU's SRAM and HBM
85%VerifiedFlashAttention reduces GPU memory usage from being proportional to the square of the sequence length to a linear relationship by redesigning the memory access pattern of the attention computation.
80%VerifiedFlashAttention improves inference speed and maintains mathematical equivalence with standard attention computation by reordering the computation sequence and leveraging the GPU's SRAM high-speed cache to reduce memory read/write operations.
80%VerifiedTogether AI has built an optimization system in the field of inference optimization based on high-performance operators such as FlashAttention and FlashDecoding, and has its own self-developed inference engine.
75%VerifiedFlashAttention was proposed by Tri Dao et al. in 2022.
75%VerifiedFlashAttention reduces HBM access counts from O(N²) to O(N²/M) through tiling and recomputation strategies, achieving a 2–4× speedup in long-sequence scenarios.
75%VerifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
65%Verified自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²
65%VerifiedFlashAttention (2022) reduces memory usage from O(n²) to O(n) through IO-aware tiled computation and recomputation strategies, without changing mathematical equivalence.
65%VerifiedOptimization techniques such as Flash Attention and sparse attention can compress memory complexity to near-linear.
65%VerifiedFlashAttention fuses the multi-step computation of the attention mechanism into a single CUDA kernel, reducing memory usage from O(N²) to O(N) and achieving a 2–4x speed improvement.
65%UnverifiedFlashInfer 是专为 LLM 推理设计的高性能 CUDA 算子库
90%Unverified上下文窗口扩展得益于ALiBi位置编码、RoPE的外推扩展、Flash Attention和Ring Attention等技术突破
85%UnverifiedFlashAttention reduces the number of reads and writes to HBM by retaining intermediate results in SRAM as much as possible, thereby breaking through the memory bandwidth bottleneck to achieve speedup.
60%UnverifiedGPUs adopt a SIMT (Single Instruction, Multiple Threads) architecture, which requires a high degree of regularity in data and tasks.
60%UnverifiedFlashAttention fuses the QKV computation, softmax, and output projection of the attention mechanism into a single kernel, reducing memory complexity from O(N²) to O(N) and achieving 2–4x end-to-end speedup.
60%