AI evals, explained
The most AI-specific skill a product manager can own: how to measure whether an AI feature is good enough to ship, and keep it good after launch.
Evals are the single most AI-specific skill a product manager can own. An eval is a systematic test that measures whether an AI feature is good enough, on the cases that matter, before and after you ship. If you cannot measure quality, you cannot ship AI responsibly, and you cannot tell whether a change made things better or worse.
Why normal QA is not enough
Traditional software is deterministic: the same input gives the same output, so a test either passes or fails. AI features are probabilistic, so the same input can give different outputs, and quality is a matter of degree. That is why AI products need evals, a way to score quality across many realistic cases, rather than a simple pass or fail check.
How to build an eval, in four steps
- Collect real cases: gather a representative set of inputs your users actually send, including the hard and weird ones.
- Define good: decide what a good answer looks like for each case, sometimes with an ideal answer, sometimes with a rubric.
- Score: measure the model output against that standard, by rules, by human review, or increasingly by using another model as a judge.
- Set a bar: decide the quality threshold you will not ship below, and monitor it over time.
Offline and online evals
Offline evals run before launch against a fixed set of cases, so you can compare versions safely. Online evals watch real usage after launch, catching drift and edge cases the offline set missed. You need both: offline to decide whether to ship, online to know whether it is still working.
What good product managers do with evals
- Treat the eval set as a living asset that grows as new failures appear.
- Use evals to turn subjective debates about quality into measurable decisions.
- Tie the quality bar to the user cost of being wrong, which varies by feature.
- Watch for drift after launch and re-run evals when the model or data changes.
Frequently asked questions
What are evals in AI products?
An eval is a systematic test that measures whether an AI feature is good enough on the cases that matter, before and after launch. Because AI outputs are probabilistic, quality is a matter of degree, so evals score quality across many realistic cases rather than giving a simple pass or fail.
Why can't you use normal QA for AI features?
Traditional software is deterministic, so a test passes or fails. AI features can give different outputs for the same input, and quality varies by degree. That is why they need evals, which score quality across many cases, instead of a single pass or fail check.
How do you build an eval?
Collect a representative set of real inputs including hard cases, define what a good answer looks like, score the model output against that standard by rules, human review or a model judge, and set a quality bar you will not ship below. Then grow the set as new failures appear.
What is the difference between offline and online evals?
Offline evals run before launch against a fixed set of cases so you can compare versions safely and decide whether to ship. Online evals watch real usage after launch to catch drift and edge cases the offline set missed. Mature AI products use both.
Who owns evals, the PM or the engineer?
Both contribute, but the product manager should own the definition of good and the quality bar, since those are product decisions tied to user value. Engineers and data scientists help build and run the scoring. Owning evals is one of the clearest ways a PM adds value on an AI product.