Foundations · Foundational · Intermediate

Activation Function

The non-linear function applied after each layer that lets a network represent more than a linear mapping.

What Activation Function is

Without non-linearity, stacking layers collapses into a single linear transformation. Activations are what make depth meaningful.

How it works

ReLU zeroes negatives and is cheap; GELU and SwiGLU are smoother variants common in transformers; sigmoid and tanh saturate and are now mostly confined to gates and outputs.

Why it matters

Activation choice affects gradient flow, training stability and throughput, which is why modern architectures converged on a small set of them.

Common uses

→All neural architectures
→Transformer feed-forward blocks

Strengths

✓Cheap source of expressiveness

Watch for

✓Saturating activations kill gradients

Continue exploring

Neural Network Transformer Backpropagation