You cannot debug what you did not record
When someone reports a wrong answer, the questions are what was asked, what was retrieved, which prompt and model were used, and what came back. All four have to have been recorded at the time.
Reproducing an AI failure after the fact is nearly impossible without the trace. Recording it costs almost nothing and it is the difference between diagnosing and guessing.
What to record per call
- The input, or a reference to it
- The retrieved context — which passages, from which documents
- The prompt version and model version
- The raw output, before any post-processing
- Latency, tokens and cost
- Any validation failure or retry
Metrics that matter
| Metric | Watch for |
|---|---|
| Latency percentiles | Degradation, not just averages |
| Error and retry rate | Provider or prompt problems |
| Validation failure rate | Output drifting from the schema |
| Refusal rate | Retrieval broken |
| Cost per request | Context growing |
| Correction rate | Quality degrading |
Correlate with the originating request
A single user action may trigger retrieval, several model calls and a queued job. Without a shared identifier across all of them, tracing what happened means guessing.
Generate it once at the boundary and carry it everywhere, including into queued work.
Mind what you log
- Retrieved passages may contain personal or sensitive data
- Log references rather than content where the material is sensitive
- Apply a retention period automatically
- Remember these logs are in scope for data requests
- Restrict who can read them