Infrastructure · Established · Advanced

KV Cache

The stored key and value tensors from previous tokens that let a model generate each new token without recomputing the whole sequence.

What KV Cache is

During generation the model would otherwise redo attention over the entire prompt for every token. The KV cache keeps those intermediate tensors in memory instead.

How it works

Each generated token appends to the cache. Memory grows with context length, batch size and model width, which makes the cache the main constraint on how many concurrent long-context users a GPU can serve. Grouped-query attention and paged attention reduce and manage that footprint.

Why it matters

It explains why long context is expensive to serve and why the first token is slow while subsequent tokens are fast.

Common uses

→Serving optimisation
→Throughput and cost planning
→Long-context capacity design

Strengths

✓Dramatic speedup after the first token

Watch for

✓Dominates GPU memory at long context

Continue exploring

Inference Attention Mechanism Context Window AI Latency