Explained · Engineering
AI evaluations (evals)
Evals are the test suite for an AI system: a fixed set of questions with known good answers, scored automatically every time something changes. Without them, you are guessing whether a new model or prompt actually made things better. Teams with sharp evals ship improvements; teams without them ship vibes.
Where it breaksAn eval measures what it measures, and a system tuned against it gets better at the test rather than the job. Public benchmarks leak into training data, so a high score can mean the answers were memorised. And the questions people most want graded — was that summary any good — have no single right answer, so teams grade them with another model and inherit its biases.
57 subjects breadth of MMLU, the benchmark most model launches still quoteHendrycks et al., "Measuring Massive Multitask Language Understanding" (ICLR 2021) · 2021-01-12
Why software testing does not transfer
Ordinary code is deterministic: the same input gives the same output, so a test asserts equality and you are done. A model gives a slightly different answer every time, and two different answers can both be correct. So an eval cannot check for one exact string. It checks a property instead — does the output contain the right figure, does the JSON parse, does the generated code pass its own tests, does a grader judge it as good as the reference. That shift has a consequence people underestimate: you are no longer testing, you are measuring. A single run tells you very little, because the same system scores differently on Tuesday. What an eval gives you is a number stable enough to compare two versions of your system, which is the only comparison that matters.
What a real eval set looks like
It is a fixed list of cases drawn from your own traffic, not from a leaderboard. Each case has an input, a note on what a good answer must contain, and a scoring method — exact match where the answer is a fact, a parser where the answer is structured, a code run where the answer is code, and a model asked to grade against a rubric only where nothing cheaper will do. The set grows the way a bug tracker grows: every time the system fails a user, that failure becomes a case, so the eval accumulates exactly the mistakes your product actually makes. Fifty well-chosen cases from your own logs beat any public benchmark for deciding whether to ship, because they are the distribution you serve.
Where it breaks
Three ways, and all three are quiet. The first is optimising the measure: tune prompts against a fixed set for long enough and you improve the score without improving the product, because you have been fitting the test. The second is contamination — public benchmarks end up in training data, so a strong score on a well-known set may mean the model saw the answers, which is why launch-day benchmark numbers are the weakest evidence in the industry. The third is the grader. Using a model to score another model's writing is often the only practical option, and it imports that grader's preferences: it rewards length, confidence and its own house style, so your system drifts towards whatever the judge likes. Spot-check graded cases by hand regularly, or the number slowly stops meaning anything.
The decision it buys you
The point of an eval is not a score to publish; it is permission to change things. Without one, every prompt edit and model swap is a coin flip dressed up as an improvement, and teams end up frozen — nobody upgrades the model because nobody can prove the upgrade is safe. With one, a cheaper model becomes a testable proposition rather than a risk, which is precisely what unlocks routing, distillation and every other cost move. That is the honest reason to build the eval set first: it is the instrument that lets you spend less, and you cannot safely spend less without it.