Moyan AI Training Institution LogoMoyan AI

Practice · Established · Intermediate

Dropout

A regularisation method that randomly zeroes a fraction of units during training so the network cannot rely on any single pathway.

What Dropout is

Dropout effectively trains an ensemble of sub-networks that share weights, making learned features more redundant and robust.

How it works

During each training step a probability p decides which activations are dropped; at inference all units are active with outputs rescaled to keep expected magnitudes consistent.

Why it matters

It became a default component of deep networks because it is simple, cheap and effective, though modern transformer training often uses lower rates alongside other techniques.

Common uses

  • →Convolutional and fully connected networks
  • →Transformer training
  • →Small-data fine-tuning

Strengths

  • ✓Trivial to implement
  • ✓Strong regularisation effect

Watch for

  • ✓Slows convergence
  • ✓Rate needs tuning

Continue exploring

More in this collection

Browse all AI Concepts