Infrastructure · Established · Beginner

Response Streaming

Sending a model's output token by token as it is generated instead of waiting for the full response.

What Response Streaming is

Streaming changes the felt experience of an assistant. Users start reading immediately, and can stop a wrong answer early rather than paying for the whole generation.

How it works

Servers emit incremental chunks over server-sent events or websockets; clients append them to the UI. Tool-calling flows stream structured events alongside text.

Why it matters

It is the single cheapest perceived-performance improvement available in an AI product.

Common uses

→Chat interfaces
→Live transcription
→Agent progress display

Strengths

✓Much better perceived speed
✓Allows early cancellation

Watch for

✓Harder to validate output before display
✓Complicates structured parsing

Continue exploring

AI Latency Inference AI User Experience