Practice · Established · Intermediate

Dropout

A regularisation method that randomly zeroes a fraction of units during training so the network cannot rely on any single pathway.

What Dropout is

Dropout effectively trains an ensemble of sub-networks that share weights, making learned features more redundant and robust.

How it works

During each training step a probability p decides which activations are dropped; at inference all units are active with outputs rescaled to keep expected magnitudes consistent.

Why it matters

It became a default component of deep networks because it is simple, cheap and effective, though modern transformer training often uses lower rates alongside other techniques.

Common uses

→Convolutional and fully connected networks
→Transformer training
→Small-data fine-tuning

Strengths

✓Trivial to implement
✓Strong regularisation effect

Watch for

✓Slows convergence
✓Rate needs tuning

Continue exploring

Neural Network Overfitting