What to Watch on a Live AI System

Uptime tells you almost nothing

The characteristic AI failure is a system that is up, fast and answering badly. Every conventional monitor is green while the output degrades.

So the monitoring has to be about the output, which means deciding what normal output looks like before launch.

Five metrics that matter

Alert on absence

Zero output where output is normal is the failure that runs longest, because nothing errors. A job that processes nothing looks identical to a job with nothing to process.

So we alert in both directions: too few and too many.

Scheduled evaluation as a canary

Run the fixed evaluation set on a schedule, not only on deploys. A provider-side model change will show up there before any user notices.

Alert on a drop beyond normal variation, and record the result over time so the trend is visible rather than the single number.

Route alerts to someone who can act

• Quality alerts to the business owner, not only to engineers
• Cost alerts to whoever owns the budget
• Failure alerts to whoever can restart or disable
• A weekly summary even when nothing fired, so the system stays visible

An alert nobody reads is worse than no alert, because it creates the belief that something is being watched.

Frequently asked questions