Unverified60% confidenceFactExact time
Qwen3 27B combines three linear attention layers with one full attention layer in a recurrent structure at a ratio of 3:1.
2
Sources
60%
Confidence
Long-term
Relevance
9/2/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedQwen3-Next and Kimi Linear both adopt a 3:1 ratio of linear attention to softmax attention.78% similarUnverifiedIQuest-Q1 employs a mixed ratio of 3 layers of sliding window attention combined with 1 layer of full attention, with only 25 out of 88 layers expanding the KV cache as context grows.75% similarUnverifiedK3 uses a mixture of linear attention layers and standard attention layers to seek a balance between accuracy and efficiency.73% similarUnverifiedFour-dimensional tensors typically appear in multi-head attention mechanisms, where the dimension is evenly divided into several parts and then arranged and processed as a four-dimensional tensor.73% similarVerifiedQwen3.5 uses a hybrid attention architecture that mixes Gated Delta Net (a linear attention variant) with Gated Attention (standard attention).72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/843030API
curl https://kongchang.com/api/v1/knowledge/claims/843030MCP
get_claim(id=843030)