Uptime tells you almost nothing
The characteristic AI failure is a system that is up, fast and answering badly. Every conventional monitor is green while the output degrades.
So the monitoring has to be about the output, which means deciding what normal output looks like before launch.
Five metrics that matter
| Metric | Alert when | What it usually means |
|---|---|---|
| Correction rate | Up 50% week on week | Input drift or a prompt regression |
| Refusal rate | Up sharply | Retrieval has broken, not the model |
| Confidence distribution | Shifting down | Input no longer resembles the test cases |
| Cost per item | Up 30% | Longer prompts, retries, or a loop |
| Output volume | Outside the normal band | Silent failure upstream |
Alert on absence
Zero output where output is normal is the failure that runs longest, because nothing errors. A job that processes nothing looks identical to a job with nothing to process.
So we alert in both directions: too few and too many.
Scheduled evaluation as a canary
Run the fixed evaluation set on a schedule, not only on deploys. A provider-side model change will show up there before any user notices.
Alert on a drop beyond normal variation, and record the result over time so the trend is visible rather than the single number.
Route alerts to someone who can act
- Quality alerts to the business owner, not only to engineers
- Cost alerts to whoever owns the budget
- Failure alerts to whoever can restart or disable
- A weekly summary even when nothing fired, so the system stays visible
An alert nobody reads is worse than no alert, because it creates the belief that something is being watched.