The failure mode is muting, not missing
Monitoring guidance tends towards watching everything: every input distribution, every output, every performance metric. Implemented literally on reasonable data volumes, this produces alerts constantly.
The team mutes the channel, and monitoring now provides negative value - the appearance of oversight without any. The design goal is a small number of alerts that each demand a decision.
Statistical significance is the wrong trigger
With enough records, tiny differences become statistically significant. A drift test on a million rows will detect a shift that changes no prediction meaningfully.
Alert on effect size rather than significance, and set the size from what actually matters. The question is not 'has the distribution changed' but 'has it changed enough to alter decisions or degrade accuracy'.
A short list worth watching
| Signal | Why | Alert when |
|---|---|---|
| Prediction distribution | Cheapest early warning; needs no outcomes | Shifts beyond its normal range and stays |
| Top few input features | Where drift usually starts | A material shift persists across periods |
| Missing-value rates | Detects broken pipelines fast | Any sudden change - usually a bug |
| Accuracy on outcomes | The real measure | Sustained decline against baseline |
| Fallback and error rates | Model not running at all | Any sustained increase |
The missing-value row is worth singling out. A sudden jump in blanks almost always means an upstream change, and it is the fastest-acting, most actionable alert on the list.
Require persistence
Single-period alerts fire on ordinary variation. Requiring a condition to hold for two or three consecutive periods removes most false alarms at the cost of a short delay.
For genuinely fast-moving risks keep one immediate alert - a pipeline failure or an error rate spike - and let everything else require persistence. That split keeps the urgent channel urgent.
Make the alert say what to do
An alert reading 'feature drift detected on customer_tenure' tells nobody what to do at 7am. An alert should name the metric, the threshold crossed, how long it has held, the likely causes and the expected next step.
- Write the runbook before enabling the alert.
- Name an owner for each alert - not a team address.
- Record what was done each time it fires; after a few, patterns emerge.
- Review the alert set quarterly and delete the ones that never led to action.
That last step is the one nobody does, and it is what keeps monitoring useful over years rather than months.
An alert that never changes anyone's behaviour is a notification, and it will be muted.