Unverified60% confidenceFactExact time
GQA allows multiple query heads to share the same set of KV heads; LLaMA-3 70B uses 8 KV heads corresponding to 64 query heads, making the KV Cache only 1/8 that of standard MHA.
2
Sources
60%
Confidence
Long-term
Relevance
9/10/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.72% similarUnverifiedGQA让多个查询头共享同一组键值头，Llama 2 70B 和 Llama 3 系列均采用了 GQA71% similarUnverifiedGQA reduces the size of the KV Cache by having multiple Query heads share the same set of Key-Value heads.68% similarUnverifiedMQA令所有查询头共享同一组Key-Value以压缩KV Cache65% similarVerifiedGrouped Query Attention (GQA) and Multi-Query Attention (MQA) reduce KV cache memory usage during inference by having multiple query heads share the same group of key/value heads, and are standard optimizations in models such as Llama 3 and Mistral.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/887577API
curl https://kongchang.com/api/v1/knowledge/claims/887577MCP
get_claim(id=887577)