Anecdote is the default and it is terrible
Without a measurement set, AI systems are improved by whoever complained most recently. Someone reports a bad answer, the prompt is adjusted, something else degrades, and nobody notices because nobody is measuring.
The fix is unglamorous: a fixed set of cases you re-run on every change.
Building the set
- Take real cases, not invented ones. Real inputs are messier and that is the point.
- Include the awkward ones deliberately — the ambiguous, the incomplete, the unusual.
- Agree the correct answer with someone who knows the domain, in writing.
- Include cases where the right answer is “I do not know”, because refusal behaviour needs testing too.
- Aim for a few hundred, which is enough for most business features.
The disagreements that arise while agreeing correct answers are themselves valuable. If two experts disagree on 20% of cases, no system will exceed 80% agreement and you have learned the ceiling before building.
Choose metrics that reflect consequence
Overall accuracy hides the failures that matter. A system that is 95% accurate but wrong on your highest-value cases is worse than one that is 90% accurate uniformly.
- Measure by category, not just overall
- Weight by business consequence where errors differ in cost
- Track false positives and false negatives separately — they usually cost different amounts
- Report straight-through rate, not just field accuracy, where human review is involved
Use a model to grade, carefully
For subjective outputs, having a model evaluate against criteria is a practical approach and needs validating: check its grades against human grades on a sample before trusting them.
Where the criteria are objective, prefer deterministic checks. A model grading whether a date was extracted correctly is an unnecessary source of noise.
Watch for the trap of overfitting to the set
If you tune repeatedly against the same cases, you will eventually optimise for them rather than for the real distribution. Hold back a portion you do not tune against, and refresh the set periodically with new real cases.
Also re-examine the set when the world changes: new document formats, new product lines, new customer types.
Run it automatically
The evaluation should run on every change, without anyone remembering. Manual evaluation happens twice and then stops, at which point you are back to anecdote.