Practice · Fast-moving · Advanced

Model Distillation

Training a smaller student model to reproduce the behaviour of a larger teacher model.

What Model Distillation is

Distillation transfers capability rather than data. The student learns from the teacher's outputs or internal distributions, often reaching most of the quality at a fraction of the size.

How it works

The student is trained on teacher-generated responses, soft probability targets, or intermediate representations, usually on a task-focused prompt distribution rather than the whole internet.

Why it matters

It is how production teams get acceptable quality at low latency and cost, and how many compact open models are produced. Note that distilling from a commercial API may breach its terms.

Common uses

→Compact task-specific assistants
→On-device models
→Latency-critical features

Strengths

✓Big cost and speed wins
✓Retains most task quality

Watch for

✓Narrower generalisation
✓Licensing and terms-of-service constraints

Continue exploring

Small Language Model Quantization Fine-Tuning Synthetic Data