Conventional testing does not transfer
You cannot assert that an answer equals an expected string. Two correct answers can be worded entirely differently, and a subtly wrong one can be worded almost identically to a right one.
So evaluation is about judged correctness across a set, not exact matching on a case.
Build the set from real questions
- Take a hundred real questions from your support log or inbox
- Have someone who knows write the correct answer for each
- Include the ones with no good answer, where refusal is correct
- Include ambiguous ones, where asking back is correct
- Hold back a portion you never tune against
The refusal cases matter as much as the answerable ones. A system that never refuses will invent, and only the refusal cases catch that.
Run it automatically
On every prompt change, every model change and on a schedule. Manual evaluation happens once and then stops, which is the same as not having it.
Record the score over time so a gradual decline is visible rather than being discovered as a complaint.
Sample live output too
- Twenty live answers a week, reviewed by someone who knows
- Every reported wrong answer, added to the evaluation set
- Every refusal, checked — is the content missing or is retrieval broken?
- The unanswered-question log, reviewed monthly
Judge on what matters
| Dimension | Question |
|---|---|
| Correctness | Is the claim true? |
| Grounding | Does the cited source support it? |
| Completeness | Did it answer the whole question? |
| Refusal | Did it decline when it should have? |
| Tone | Would you be happy for a customer to read this? |