Evaluating Whether an AI Feature Is Good Enough
Last updated:
“It seems good” is not a measurement
The most common way an AI feature goes wrong is that nobody can tell it has. Quality drifts — a provider updates a model, your documents change, the input distribution shifts — and the first signal is a complaint.
The fix is unglamorous: a fixed set of real examples with known-correct answers, run automatically, with the score recorded every time.
Building the set
- Take 50–200 real examples, sampled across the range you actually see
- Have someone who knows the domain write the correct answer for each
- Include the hard cases deliberately — the ambiguous, the malformed, the unusual
- Include a handful that should be refused or escalated
- Store it in version control alongside the code
Two days of work, and it is the difference between engineering and guessing. Every AI project we run has one before any prompt is written.
If you cannot say what the right answer is for fifty real examples, you do not yet understand the task well enough to automate it.
Choosing a metric that matches the stakes
| Task | Metric | Why |
|---|---|---|
| Classification | Per-class precision and recall | Overall accuracy hides a class you always get wrong |
| Extraction | Field-level exact match | One wrong field spoils the record |
| Retrieval | Was the right passage in the top results | The answer cannot be right if the source was missed |
| Drafting | Human accept-without-edit rate | The only thing that correlates with time saved |
| Agents | Task completed correctly, end to end | Per-step accuracy compounds misleadingly |
For anything with asymmetric cost, weight the metric accordingly. Missing a fraudulent transaction is not the same as flagging a legitimate one, and a single accuracy figure obscures the difference.
Regression testing on every change
The evaluation set runs whenever the prompt changes, the model changes, the retrieval changes or the provider ships an update. A drop beyond a threshold blocks the deployment, exactly like a failing unit test.
This catches the two failure modes that would otherwise be invisible: a prompt improvement that helps one case and breaks four, and a provider update that quietly changes behaviour on your edge cases.
Watching production, not just the test set
- Sample a percentage of live outputs for human review weekly
- Track the confidence distribution — drift shows up here first
- Track how often users edit, reject or escalate
- Alert on rate changes rather than on individual events
Human review of live samples is what finds the failures your evaluation set never imagined. Those cases then get added to the set, which is how it stays useful rather than becoming a museum of last year's problems.
Deciding when it is good enough
Good enough is defined by what happens to the errors, not by a number in the abstract. With a human review step, a lower bar is fine because mistakes are caught. Fully automated, you need the residual error rate to be genuinely tolerable at your volume.
So the honest way to state a threshold is: at this accuracy and this volume, we will produce roughly this many errors a week, and here is who handles them. If that sentence is acceptable when written down, the feature is good enough to ship.
Frequently asked questions
How big should an evaluation set be?
Who should write the correct answers?
How often should we re-run it?
What if the correct answer is subjective?
Need to know whether a model is good enough to ship?
We build the evaluation set first on every project. Tell us the task and we will tell you how we would measure it.