Infrastructure · Established · Intermediate

AI Latency

The delay between a request and a useful response, usually split into time to first token and generation speed.

What AI Latency is

For generative interfaces perceived speed matters more than total time, which is why streaming responses feel far faster than waiting for a completed answer.

How it works

Latency is reduced with smaller or distilled models, quantisation, prompt caching, speculative decoding, streaming, and routing simple requests to cheaper models.

Why it matters

Latency budgets decide product feasibility: voice agents and code completion have hard ceilings that batch summarisation does not.

Common uses

→Voice assistants
→Inline code completion
→Interactive chat UX

Watch for

✓Trade-off against model quality

Continue exploring

Inference KV Cache Model Distillation Edge AI Response Streaming