Inference

What Inference is

Training happens once; inference happens on every request, forever. For a successful product, cumulative inference cost dwarfs training cost.

How it works

Inference servers batch requests, cache prefixes, quantise weights and use specialised kernels to raise throughput. Key metrics are time to first token, tokens per second and cost per thousand tokens.

Why it matters

Almost every architectural decision in production AI — model size, quantisation, caching, routing between models — is an inference economics decision.

Common uses

• →Serving assistants and APIs
• →Batch document processing
• →On-device model execution