Silent corruption is the normal failure
Models rarely break with an error. Upstream something changes - a field renamed, a category recoded, a unit changed, an integration paused - and predictions continue, quietly wrong.
By the time anyone notices, weeks of decisions have been made on bad output. The defence is checking the data on every load rather than checking the model after the fact.
The checks that catch most problems
- Row count against expectation. A sudden drop usually means a partial load; a jump often means duplication.
- Schema. Expected columns present, with expected types. Catches renames and reordering.
- Missing value rates. Compare against the usual rate per column - a jump is nearly always an upstream change.
- Value ranges and categories. Numbers within plausible bounds; categorical fields containing only known values.
- Freshness. Most recent record recent enough. Catches stalled feeds, which otherwise look like quiet days.
- Uniqueness. Keys that should be unique still are. Catches duplicate loads and broken joins.
These six catch the large majority of real incidents, and none is difficult to implement.
Unseen categories deserve special mention
A new value appearing in a categorical field is one of the most damaging and least visible problems. The model has never seen it, so it either errors or silently treats it as a default - and a default that is quietly wrong is worse than a failure.
Check explicitly for values not present in training, and decide the policy: fail, route to a default with an alert, or handle as unknown. The important thing is that it is a decision rather than whatever the code happens to do.
Fail loudly, in the right direction
| Response | When appropriate |
|---|---|
| Stop the pipeline | Schema or freshness failure - the data is unusable |
| Continue with an alert | Minor range violations on non-critical fields |
| Serve the previous predictions | Where stale is better than wrong |
| Continue silently | Never |
The default in most pipelines is the last row, because that is what happens when nobody decides. Serving the previous predictions is frequently the right operational answer - yesterday's scores are usually better than scores computed from broken data.
Derive the expectations from history
Setting thresholds by hand is tedious and quickly stale. Derive them from recent history - typical row count, usual missing rate, observed range - and refresh periodically.
Keep them conservative enough not to fire constantly. A check that alerts weekly for no reason gets disabled within a month, and then it is not protecting anything. Our note on alerting without noise covers the same principle.
A pipeline that carries on regardless will happily compute predictions from nothing at all.