Core Architecture · Established · Intermediate
Grouped Query Attention Scaling
Also known as: GQA, Grouped Key-Value Attention
An attention mechanism that groups multiple query heads to share single key and value heads, balancing speed and model quality.
What Grouped Query Attention Scaling is
GQA provides near-MHA quality while reducing KV cache memory usage to near-MQA levels during decoding.
How it works
Partitions query heads into G groups, where each group shares a single K and V projection matrix across sequence positions.
Why it matters
Adopted as the standard attention architecture in modern open models including Llama 3, Mistral, and Qwen.
Common uses
- →KV cache memory reduction
- →High-concurrency LLM inference
- →Open-source foundation model design
More in this collection
Browse all AI Concepts