Infrastructure · Established · Advanced

Quantization

Storing and computing model weights at lower numerical precision to cut memory use and speed up inference.

What Quantization is

A model trained in 16-bit can often run at 8-bit or 4-bit with modest quality loss, which is what makes capable models fit on consumer GPUs and phones.

How it works

Post-training quantisation converts weights after training, sometimes with a calibration set; quantisation-aware training simulates low precision during training for better fidelity. Common formats include INT8, INT4 and various GGUF and AWQ schemes.

Why it matters

It is the main reason local and on-device AI is practical, and a standard cost lever in server deployments too.

Common uses

→Local model running on laptops
→Mobile and edge inference
→Reducing serving cost

Strengths

✓Large memory and speed gains
✓Enables on-device privacy

Watch for

✓Quality degrades at aggressive precision
✓Not all operations quantise cleanly

Continue exploring

Model Distillation Edge AI Open-Weight Model Inference