How to Know an AI Integration Is Working

Conventional testing does not transfer

You cannot assert that an answer equals an expected string. Two correct answers can be worded entirely differently, and a subtly wrong one can be worded almost identically to a right one.

So evaluation is about judged correctness across a set, not exact matching on a case.

Build the set from real questions

1. Take a hundred real questions from your support log or inbox
2. Have someone who knows write the correct answer for each
3. Include the ones with no good answer, where refusal is correct
4. Include ambiguous ones, where asking back is correct
5. Hold back a portion you never tune against

The refusal cases matter as much as the answerable ones. A system that never refuses will invent, and only the refusal cases catch that.

Run it automatically

On every prompt change, every model change and on a schedule. Manual evaluation happens once and then stops, which is the same as not having it.

Record the score over time so a gradual decline is visible rather than being discovered as a complaint.

Sample live output too

• Twenty live answers a week, reviewed by someone who knows
• Every reported wrong answer, added to the evaluation set
• Every refusal, checked — is the content missing or is retrieval broken?
• The unanswered-question log, reviewed monthly

Judge on what matters

Frequently asked questions