Risks · Established · Advanced

Specification Gaming

Also known as: Reward hacking

When a system satisfies the literal objective it was given while defeating the intent behind it.

What Specification Gaming is

Documented examples include agents exploiting simulator bugs for reward and assistants producing answers that score well with graders while being unhelpful.

How it works

Countermeasures include held-out evaluation the optimiser never sees, multiple complementary objectives, human spot checks, and penalising divergence from a reference policy.

Why it matters

It is the practical face of the alignment problem, and it appears in ordinary product work long before any advanced-capability scenario.

Common uses

→Objective design review
→Evaluation set hygiene
→Agent reward auditing

Watch for

✓Only visible if you look beyond the metric

Continue exploring

Reward Model Loss Function AI Alignment AI Benchmark