The short answer
Conventional support reproduces a bug by following steps. With AI features the same steps may produce a different result, so support needs the exact request that failed.
That requires logging enough to replay it, and a tool that lets support do so without engineering.
What support needs logged
- The request identifier the customer can quote
- The inputs and retrieved context actually used
- The model and version at the time
- The output returned
- Any error or fallback that occurred
Without the retrieved context, most investigations stall. The output is frequently reasonable given what the system actually saw, and that is the finding.
Separate the two kinds of complaint
| Complaint | Route |
|---|---|
| Feature errored or timed out | Standard engineering incident |
| Output was wrong | Replay, inspect context, categorise |
| Output was inconsistent | Explain variability, check if it is within design |
| Feature refused | Check whether the refusal was correct |
| Output was slow | Standard performance investigation |
Rows two and three are the ones support teams are not usually equipped for, and they are the majority of AI-feature tickets.
Be honest about variability
If output can vary between identical requests, say so in the documentation rather than letting customers discover it as a bug. Framed up front it is a characteristic; discovered later it feels like a defect.
Where consistency matters to a customer, explain what you do offer, such as reproducibility for a given version or the ability to pin behaviour.
Feed support back into the product
- Categorise wrong-output tickets by failure type, not just by count.
- Add the worst cases to the evaluation set.
- Look for clusters pointing at one weak area.
- Report those clusters to the product team routinely.
- Close the loop with the customer when the case is fixed.
Support tickets are the highest quality failure data a product has, and they are usually discarded after the ticket closes.