Alignment · Fast-moving · Advanced

Reward Model

A model trained to score outputs the way human raters would, used as the optimisation target during preference tuning.

What Reward Model is

The reward model is a learned stand-in for human judgement, which makes preference optimisation affordable but also makes its flaws systematic.

How it works

Pairwise human comparisons train the scorer. During policy optimisation the reward model grades candidate outputs, and a divergence penalty prevents the policy from exploiting its blind spots too aggressively.

Why it matters

Over-optimising against an imperfect reward model produces sycophancy, verbosity and confident hedging — behaviours users complain about constantly.

Common uses

→RLHF training
→Best-of-n reranking at inference
→Automated response grading

Strengths

✓Scales human preference cheaply

Watch for

✓Reward hacking
✓Encodes annotator bias

Continue exploring

Reinforcement Learning from Human Feedback AI Alignment Specification Gaming LLM Evaluation