MQA
An attention mechanism optimization technique in which all query heads share a single set of key-value heads. Compared to Grouped Query Attention (GQA), MQA has smaller KV cache overhead but incurs relatively greater model quality loss, making it a more extreme variant of GQA.
Core Facts
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
Flash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
Timeline (last 90 days)
GQA and MQA save the most in long-context, large-batch, high-concurrency inference scenarios, but the compression benefit weakens in short-sequence or small-batch scenarios, and they carry the risk of reduced expressiveness.
The core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
The architecture uses a single shared KV head (num_key_value_heads=1, i.e. MQA form), with head_dim of 512 to compensate for expressiveness.
MQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.
The compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
In MQA (Multi-Query Attention), all query heads share a single set of KVs, resulting in a smaller cache but with relatively greater quality degradation.
GQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
MQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
Flash models typically use GQA or MQA technology to compress the KV Cache by 4–8×, thereby supporting higher concurrency on the same hardware.
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
2 more timeline events
All Facts (14)
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%VerifiedGQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
65%VerifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
65%UnverifiedFlash系列模型通常采用更小的参数规模、多查询注意力(MQA)或分组查询注意力(GQA)等高效注意力机制，以及模型蒸馏技术
70%UnverifiedGQA and MQA save the most in long-context, large-batch, high-concurrency inference scenarios, but the compression benefit weakens in short-sequence or small-batch scenarios, and they carry the risk of reduced expressiveness.
50%UnverifiedThe core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
50%UnverifiedThe architecture uses a single shared KV head (num_key_value_heads=1, i.e. MQA form), with head_dim of 512 to compensate for expressiveness.
50%UnverifiedMQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.
50%UnverifiedThe compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
50%UnverifiedIn MQA (Multi-Query Attention), all query heads share a single set of KVs, resulting in a smaller cache but with relatively greater quality degradation.
50%UnverifiedGQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
50%UnverifiedMQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
50%UnverifiedFlash models typically use GQA or MQA technology to compress the KV Cache by 4–8×, thereby supporting higher concurrency on the same hardware.
50%UnverifiedGQA 通过让多个 Query 头共享同一组 Key-Value 头，在保持模型表达力的同时降低了 KV Cache 的显存占用
50%