Optimisation · Established · Advanced

Speculative decoding

Using a small fast model to draft several tokens that a larger model verifies in one pass, accelerating generation.

What Speculative decoding is

Because verification is parallel while generation is sequential, accepted drafts produce multiple tokens for roughly the cost of one step.

How it works

A draft model proposes tokens; the target model checks them and accepts the longest matching prefix, falling back to normal decoding when they diverge. Output is identical to standard decoding.

Why it matters

It is a rare optimisation that improves speed without changing output quality.

Common uses

→Latency-sensitive serving
→High-throughput batch generation

Strengths

✓Lossless speedup
✓Compatible with existing models

Watch for

✓Needs a well-matched draft model
✓Extra memory for two models

Continue exploring

Inference AI Latency Model Distillation Model Serving