Architectures · Fast-moving · Advanced

Mixture of Experts

Also known as: MoE

An architecture where a router activates only a few specialised sub-networks per token, so total parameters can grow without proportionally growing compute.

What Mixture of Experts is

Instead of every parameter processing every token, an MoE layer holds many expert feed-forward networks and routes each token to a small subset, typically one or two.

How it works

A learned gating network scores experts per token and selects the top-k. Load-balancing losses stop the router collapsing onto a few favourites, and experts are sharded across devices during training and serving.

Why it matters

It is a leading way to get frontier-scale capability at manageable inference cost, and many recent open-weight and commercial models use it.

Common uses

→Large frontier and open-weight models
→Cost-efficient serving at scale

Strengths

✓More capacity per unit of compute
✓Experts can specialise

Watch for

✓Complex distributed serving
✓High memory footprint since all experts must be resident
✓Routing instability

Continue exploring

Large Language Model Inference Model Parameters Quantization