Scaling Laws

What Scaling Laws is

Scaling laws let labs forecast the performance of a training run before committing to it, and they guided the decision to build ever larger models.

How it works

Researchers train many small models across a grid of sizes and data budgets, fit a power-law curve to the results, and extrapolate. Compute-optimal analyses showed many early models were undertrained relative to their size.

Why it matters

They explain the industry's capital-intensive trajectory, and the current debate about where returns to pure scale start to flatten.

Common uses

• →Training budget planning
• →Model size selection
• →Research roadmapping