Gradient Descent

What Gradient Descent is

Gradient descent treats training as walking downhill on a loss surface. The gradient points uphill, so the update moves the opposite way, scaled by a learning rate.

How it works

Stochastic gradient descent estimates the gradient on mini-batches instead of the full dataset. Adaptive optimisers such as Adam and AdamW maintain per-parameter step sizes and momentum, and schedules warm up then decay the learning rate.

Why it matters

Learning rate and optimiser choice dominate whether a large training run converges at all, which makes this the practical core of model training.

Common uses

• →Training every deep model
• →Fine-tuning
• →Embedding learning