Self-Attention
An attention mechanism in which each position in a sequence computes similarity with other positions, enabling the model to establish global dependencies within the same sequence. It is a core component of Transformer encoders and decoders.
Core Facts
自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²
The self-attention mechanism allows the model to simultaneously see the information of all other tokens in the input sequence when processing each token, and the computational complexity of the attention matrix is O(n²)
Timeline (last 90 days)
The Transformer is an architecture built entirely on self-attention.
Self-attention itself is permutation-invariant, so the model cannot directly learn the input order from the attention output.
Transformers access past trajectories through the attention mechanism, at the cost of quadratic memory and computation overhead.
Each Transformer block contains a self-attention layer and a position-wise feed-forward network, both surrounded by residual connections and layer normalization.
Self-Attention means that each position within a sequence computes similarity with all other positions, allowing the model to establish global dependencies within the same sequence, and it is the core of the Transformer encoder and decoder.
Tokenization is essentially table lookup and a small amount of string matching, with complexity approximately linear in sequence length, and unit cost far lower than self-attention or matrix multiplication in a Transformer.
When sequence length n grows from 2K to 128K, the attention computation volume increases approximately 4,000-fold.
The study finds that hallucinations manifest as three types of information flow distortion patterns: over-reliance on self-attention, diffuse contextual retrieval, and excessive information compression.
缩放点积注意力公式为 Attention(Q,K,V) = softmax(QK^T/√d_k)V，除以√d_k是为防止点积在高维空间中过大导致softmax进入饱和区域
All Facts (11)
自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²
65%VerifiedThe self-attention mechanism allows the model to simultaneously see the information of all other tokens in the input sequence when processing each token, and the computational complexity of the attention matrix is O(n²)
65%UnverifiedThe Transformer is an architecture built entirely on self-attention.
50%UnverifiedSelf-attention itself is permutation-invariant, so the model cannot directly learn the input order from the attention output.
50%UnverifiedTransformers access past trajectories through the attention mechanism, at the cost of quadratic memory and computation overhead.
50%UnverifiedEach Transformer block contains a self-attention layer and a position-wise feed-forward network, both surrounded by residual connections and layer normalization.
50%UnverifiedSelf-Attention means that each position within a sequence computes similarity with all other positions, allowing the model to establish global dependencies within the same sequence, and it is the core of the Transformer encoder and decoder.
50%UnverifiedTokenization is essentially table lookup and a small amount of string matching, with complexity approximately linear in sequence length, and unit cost far lower than self-attention or matrix multiplication in a Transformer.
50%UnverifiedWhen sequence length n grows from 2K to 128K, the attention computation volume increases approximately 4,000-fold.
50%UnverifiedThe study finds that hallucinations manifest as three types of information flow distortion patterns: over-reliance on self-attention, diffuse contextual retrieval, and excessive information compression.
50%Unverified缩放点积注意力公式为 Attention(Q,K,V) = softmax(QK^T/√d_k)V，除以√d_k是为防止点积在高维空间中过大导致softmax进入饱和区域
50%