Verified65% confidenceFactExact time
In the attention mechanism of the Transformer architecture, as sequence length increases, the attention scores of early tokens are diluted by subsequent tokens after softmax normalization, forming an attention sinking effect.
3
Sources
65%
Confidence
Long-term
Relevance
7/24/2026
First Seen
Sources
Harness AI工程化编程：企业级项目实战方法论
bilibili图灵学院官方7/12/2026
Related Claims
VerifiedBecause the Transformer self-attention mechanism exhibits positional bias, earlier input content has a stronger anchoring effect on subsequent generation, which is the underlying reason for the context anchoring effect of system prompts79% similarUnverified标准Softmax注意力在极长序列上存在注意力湮没（Attention Sink）问题，催生了Sliding Window Attention、RoPE外推、ALiBi等改进方案78% similarUnverified标准Attention的显存消耗随序列长度平方增长，Flash Attention将其压缩到线性增长78% similarVerifiedContext length and computational load grow in an approximately quadratic relationship, influenced by the attention mechanism.76% similarVerifiedThe Lost in the Middle effect originates from an inherent architectural bias in the Transformer attention mechanism where tokens in the middle positions receive lower attention weights, and no solution currently exists to completely eliminate it in all scenarios75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/608339API
curl https://kongchang.com/api/v1/knowledge/claims/608339MCP
get_claim(id=608339)