Architectures · Foundational · Advanced

Attention Mechanism

Also known as: Self-attention

A learned way for a model to weigh which other parts of the input matter most when representing each position.

What Attention Mechanism is

Attention answers 'what should I look at right now'. For each token the model produces a query, and compares it to keys from all other tokens to decide how much of each value to mix in.

How it works

Scores are the scaled dot products of queries and keys, softmaxed into weights that combine the values. Multiple heads run in parallel to capture different relationship types. Efficient implementations such as FlashAttention reduce memory traffic, and grouped-query attention trims the key-value cache for faster serving.

Why it matters

Attention is the mechanism that lets a model resolve pronouns, track entities across paragraphs and follow instructions stated far earlier in the prompt.

Common uses

→All transformer models
→Cross-attention between text and images
→Retrieval-augmented conditioning

Strengths

✓Direct access to distant context
✓Interpretable to a limited degree via attention maps

Watch for

✓Quadratic cost in sequence length
✓Attention maps are not reliable explanations

Continue exploring

Transformer Context Window KV Cache State Space Model