Building an LLM Evaluation Harness That Actually Catches Regressions
Without scored evaluation, prompt engineering is superstition. How to build a golden set, choose metrics that mean something, and wire it into CI.
Change a prompt, and three behaviours you were not thinking about change with it. Without a scored evaluation set you will not notice until a user does. An evaluation harness is the difference between engineering an AI system and adjusting it hopefully.
Start with failures, not coverage
The instinct is to assemble a broad, representative dataset. Better to start with everything that has already gone wrong: the answers users complained about, the outputs support had to correct, the cases that embarrassed you in a demo. Thirty real failures are worth more than three hundred synthetic examples, because they encode the specific ways your system breaks.
Your evaluation set should be a museum of every mistake the system has made.
Pick metrics that survive scrutiny
Exact-match works for extraction and classification. For generative answers you need something else, and the practical options are a rubric scored by a stronger model, similarity to a reference answer, or a set of assertions the output must satisfy. Assertions are underrated — 'contains a citation', 'does not exceed 200 words', 'refuses when the context is missing' — because they are unambiguous and cheap to run.
The judge model needs judging
Using a model to grade a model works, with a caveat people skip: validate the judge against human ratings on a sample first. If the judge agrees with your reviewers 90% of the time, its scores are useful. If it agrees 60% of the time you are measuring the judge, not the system. Check this once, then re-check whenever you change the judge.
Run it in CI, block on regressions
Evaluation that runs when someone remembers is evaluation that does not run. Wire the suite into the pipeline so a prompt change is a pull request with a score attached, and set a threshold that fails the build. Because model calls cost money and time, run a fast subset on every commit and the full set nightly and before release.
Track scores over time
A single score tells you little; the trend tells you everything. Store every run with the prompt version, model version and parameters, so when quality drops you can see exactly what changed. This also catches the failure mode nobody plans for: the provider silently updating the model behind an endpoint you did not pin.
Feed production back in
Every real failure becomes a new case. A thumbs-down in the interface, a support escalation, a manual correction — each should land in a queue that a human triages into the evaluation set weekly. Over a year this compounds into the most valuable asset in the system, and the one a competitor cannot copy.
