Unverified50% confidenceFactExact time
IQuest-Q1 employs a mixed ratio of 3 layers of sliding window attention combined with 1 layer of full attention, with only 25 out of 88 layers expanding the KV cache as context grows.
1
Sources
50%
Confidence
Long-term
Relevance
10/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCompressed absorbing attention assigns each layer one of three modes—full, re-indexing, or reuse—to achieve inter-layer KV cache sharing.76% similarVerifiedThe Multi-Query Attention technique allows multiple attention heads to share a single set of Key and Value projection matrices, compressing KV cache memory requirements to 1/H of the original, at the cost of a slight reduction in model expressiveness.75% similarUnverifiedFour-dimensional tensors typically appear in multi-head attention mechanisms, where the dimension is evenly divided into several parts and then arranged and processed as a four-dimensional tensor.75% similarUnverifiedQwen3 27B combines three linear attention layers with one full attention layer in a recurrent structure at a ratio of 3:1.75% similarUnverifiedThe computational complexity of standard self-attention is quadratic with respect to sequence length, and achieving a context of one million tokens requires introducing techniques such as sparse attention, sliding window attention, and KV Cache compression.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/972270API
curl https://kongchang.com/api/v1/knowledge/claims/972270MCP
get_claim(id=972270)