Loss Function

What Loss Function is

The loss encodes the objective. Cross-entropy for classification and next-token prediction, mean squared error for regression, contrastive losses for embeddings, and preference losses for alignment.

How it works

Loss is computed per example, averaged over the batch, and differentiated to produce gradients. Auxiliary terms add regularisation or multi-task objectives.

Why it matters

A model does what its loss rewards, not what the team intended. Most surprising behaviour traces back to an objective that was subtly the wrong one.

Common uses

• →Every training run
• →Multi-objective tuning
• →Alignment objectives