Unverified50% confidenceFactExact time
Attention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformers assign a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
1
Sources
50%
Confidence
Long-term
Relevance
8/31/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSora adopts the Diffusion Transformer (DiT) architecture, introducing the Transformer's global attention mechanism into the diffusion framework.66% similarUnverified在Transformer架构中，MLP层承担了大量的信息处理工作64% similarUnverifiedTransformers库集成了GPTQ、AWQ、bitsandbytes等量化技术和Flash Attention加速方案64% similarUnverifiedMegatron-LM was the first to systematically propose a tensor parallelism scheme for Transformers, splitting attention heads and FFN layers across different devices.63% similarUnverifiedThe Transformer architecture uses a Self-Attention mechanism to achieve efficient parallel processing of sequential data, replacing RNN/LSTM architectures.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/829111API
curl https://kongchang.com/api/v1/knowledge/claims/829111MCP
get_claim(id=829111)