Alignment · Fast-moving · Advanced

Reinforcement Learning from Human Feedback

Also known as: RLHF

A tuning method where human preference comparisons train a reward model, which then guides reinforcement learning of the assistant's behaviour.

What Reinforcement Learning from Human Feedback is

RLHF is the bridge between a model that predicts plausible text and an assistant that is helpful, honest and refuses harmful requests. Humans rank candidate responses, and those rankings become the training signal.

How it works

First a base model is supervised fine-tuned on demonstrations. Then annotators compare pairs of outputs; a reward model learns to score responses in line with those comparisons. Finally the policy is optimised against the reward model, usually with PPO or a direct preference optimisation variant, with a penalty for drifting too far from the base model.

Why it matters

Almost every commercial chat assistant went through some form of preference tuning. It is the main practical lever teams have for behaviour, tone and refusal boundaries.

Common uses

→Instruction-following assistants
→Safety refusal behaviour
→Reducing verbosity and sycophancy
→Domain-specific response style

Strengths

✓Directly optimises what people actually prefer
✓Works without exhaustive rule-writing

Watch for

✓Preferences of annotators become the model's values
✓Can encourage agreeable but inaccurate answers
✓Expensive human labelling loop

Continue exploring

AI Alignment Reward Model Constitutional AI Fine-Tuning Direct Preference Optimization