A/B Testing for AI

What A/B Testing for AI is

Offline metrics tell you whether a model is more accurate; A/B tests tell you whether that accuracy changes user or business outcomes, which is often a different question.

How it works

Users are randomly assigned to control and treatment, a primary metric and minimum detectable effect are fixed in advance, and results are read only after the planned sample size to avoid peeking bias.

Why it matters

For generative features especially, offline evaluation is weak, so controlled online measurement is the only trustworthy evidence of value.

Common uses

• →Comparing model versions
• →Prompt and UX variants
• →Recommender ranking changes