Learning Rate

What Learning Rate is

Too high and training diverges or oscillates; too low and it crawls or settles in a poor region. It is usually the first hyperparameter to tune.

How it works

Schedules typically warm up from near zero, hold a peak, then decay by cosine or linear rules. Adaptive optimisers modulate per-parameter effective rates on top of the schedule.

Why it matters

In large model training, the learning-rate schedule is one of the few knobs that reliably decides success or failure at a cost of millions.

Common uses

• →Pretraining schedules
• →Fine-tuning with much lower rates
• →LoRA adapter training