GQA
An attention mechanism optimization technique in which multiple query heads share a single set of key and value heads. This approach significantly reduces the GPU memory footprint of the KV Cache while preserving model performance close to that of standard multi-head attention.
Core Facts
Paged Attention (PagedAttention) is the core technology of vLLM, managing KV cache in blocks, while Flash Attention optimizes the memory access pattern of attention computation.
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
Timeline (last 90 days)
The developer manually implemented four core LLaMa architecture components—RoPE, GQA, RMSNorm, and SwiGLU—without relying on an existing training framework.
GQA and MQA save the most in long-context, large-batch, high-concurrency inference scenarios, but the compression benefit weakens in short-sequence or small-batch scenarios, and they carry the risk of reduced expressiveness.
The core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
The industry commonly uses architectural approaches such as GQA and MLA to compress the KV cache.
Long contexts significantly increase the GPU memory footprint of the KV Cache, which is often mitigated in engineering through grouped-query attention (GQA), KV Cache quantization or paged management.
Models using GQA (grouped-query attention) or MLA (multi-head latent attention) can significantly compress the KV Cache, which is the key architectural prerequisite for running long contexts on small VRAM.
Grouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA).
Llama 3.1 70B uses 8 KV heads instead of 64 attention heads, reducing the KV cache size to about one-eighth of the original.
On-device deployment often comes with mechanisms such as KV cache quantization (e.g., compressing to 4 or 8 bits), grouped-query attention (GQA), and sliding-window or sparse attention.
MQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.
40 more timeline events
All Facts (20)
Paged Attention (PagedAttention) is the core technology of vLLM, managing KV cache in blocks, while Flash Attention optimizes the memory access pattern of attention computation.
80%VerifiedGQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%VerifiedGQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
65%VerifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
65%VerifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
65%UnverifiedGrouped Query Attention (GQA) strikes a balance between inference efficiency and model quality.
70%UnverifiedFlash系列模型通常采用更小的参数规模、多查询注意力(MQA)或分组查询注意力(GQA)等高效注意力机制，以及模型蒸馏技术
70%Partially VerifiedGPT-OSS adopts a MoE (Mixture of Experts) architecture combined with Grouped Multi-Query Attention (GQA).
65%UnverifiedLFM2.5-24B-A2B uses a 40-layer architecture: 30 double-gated short convolution blocks + 10 GQA attention layers, 64 experts per MoE block, top-4 routing, a 131,072 token context, and a 128K vocabulary.
60%UnverifiedMainstream open-source models such as Llama 3 and Qwen2/3 have already adopted GQA.
60%UnverifiedGQA allows multiple query heads to share the same set of KV heads; LLaMA-3 70B uses 8 KV heads corresponding to 64 query heads, making the KV Cache only 1/8 that of standard MHA.
60%UnverifiedDeepSeek 8B adopts the GQA (Grouped Query Attention) attention mechanism to reduce VRAM usage during inference.
60%UnverifiedThe developer manually implemented four core LLaMa architecture components—RoPE, GQA, RMSNorm, and SwiGLU—without relying on an existing training framework.
50%UnverifiedThe core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
50%UnverifiedGQA and MQA save the most in long-context, large-batch, high-concurrency inference scenarios, but the compression benefit weakens in short-sequence or small-batch scenarios, and they carry the risk of reduced expressiveness.
50%UnverifiedThe industry commonly uses architectural approaches such as GQA and MLA to compress the KV cache.
50%UnverifiedLong contexts significantly increase the GPU memory footprint of the KV Cache, which is often mitigated in engineering through grouped-query attention (GQA), KV Cache quantization or paged management.
50%UnverifiedModels using GQA (grouped-query attention) or MLA (multi-head latent attention) can significantly compress the KV Cache, which is the key architectural prerequisite for running long contexts on small VRAM.
50%UnverifiedGrouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA).
50%UnverifiedLlama 3.1 70B uses 8 KV heads instead of 64 attention heads, reducing the KV cache size to about one-eighth of the original.
50%