Knowing Whether a Change Made Things Better
Last updated:
You cannot improve what you cannot measure
Prompt engineering without an evaluation set is adjusting text and forming impressions. It feels productive and it produces regressions nobody notices.
The single most valuable artefact in an AI project is a hundred real cases with agreed correct answers. Everything else follows from having it.
Building the set
- Take real cases from your actual data or support log
- Have someone who knows write the correct answer for each
- Include cases where refusal is correct — questions with no good answer
- Include ambiguous cases where asking back is correct
- Hold back a portion you never tune against
How to score
| Task | Scoring |
|---|---|
| Classification | Exact match against the label |
| Extraction | Field by field, exact or normalised |
| Answering | Model-assisted grading, with human-checked samples |
| Refusal cases | Did it decline when it should? |
| Retrieval | Was the right passage in the top results? |
Extraction and classification score cleanly. Free-text answers need model-assisted grading with periodic human verification of the grader itself.
Run it automatically
- On every prompt change
- On every model change
- On a schedule, to catch provider-side changes
- With the score recorded over time, so trends are visible
- Blocking a deployment if it drops beyond normal variation
Refresh it
A set built in January describes January. Add new real cases monthly, especially failures, and retire cases that no longer represent your work.
Without refreshing, you eventually optimise for a test set that no longer resembles reality, which is worse than having none.
Frequently asked questions
How many cases do we need?
Who writes the correct answers?
Can a model grade itself?
How long does building one take?
Adjusting prompts and hoping?
An evaluation set turns that into a measurement. We can build one around a system we did not write.
Related services
What we build for problems like this one