Optimisation · Fast-moving · Intermediate

Prompt caching

Reusing the processed representation of a repeated prompt prefix so it is not recomputed on every request.

What Prompt caching is

Long system prompts, tool definitions and shared document context are identical across requests, so caching them cuts both latency and cost substantially.

How it works

Providers cache the attention state for a stable prefix for a short window; applications structure prompts so the invariant part comes first and the variable part last.

Why it matters

For agent and RAG workloads with large fixed context, it is one of the largest single cost reductions available.

Common uses

→Agents with long tool definitions
→Repeated document question answering
→High-volume assistants

Strengths

✓Big cost and latency savings
✓No quality trade-off

Watch for

✓Requires careful prompt ordering
✓Short cache lifetimes

Continue exploring

Inference KV Cache AI Latency AI API