Practice · Established · Intermediate

Hyperparameter

A configuration value chosen before training — such as learning rate, batch size or number of layers — that is not learned from data.

What Hyperparameter is

Hyperparameters shape how learning happens rather than what is learned. They interact, so tuning one in isolation is often misleading.

How it works

Search strategies include grid search, random search and Bayesian optimisation, evaluated with cross-validation or a fixed validation split. For very large runs, smaller proxy models and scaling rules guide the choices.

Why it matters

The difference between a failed and a state-of-the-art training run is frequently hyperparameters rather than architecture.

Common uses

→Model tuning
→Fine-tuning configuration
→Inference settings such as temperature

Strengths

✓Large gains without changing the model

Watch for

✓Search is compute-expensive
✓Easy to overfit the validation set

Continue exploring

Learning Rate Cross-Validation Gradient Descent Temperature