Machine Learning Incident Response: When a Model Misbehaves
Last updated:
Models fail quietly, then all at once
At 10am the customer service manager messages the ops channel: a dozen customers have complained that their discount codes were far bigger than usual. By 11am it is forty. The discount model had been assigning maximum offers to almost everyone since the previous night's data load.
Machine learning incidents have a particular shape. There is usually no error message. The system appears healthy. The first signal is a human noticing that outcomes look odd, and by then the model may have been wrong for hours or days. Having a plan written before this happens is worth a great deal.
The first hour: contain before you diagnose
The instinct is to find out why. Resist it until the damage has stopped growing.
- Confirm there is a real problem with a quick look at recent predictions rather than relying on the complaints alone.
- Switch to a safe mode: roll back to the previous model version, fall back to a rule, or route all predictions to human review.
- If none of those exist, pause the automated action entirely and accept the manual workload.
- Tell the people affected internally, such as customer service or finance, what is happening and what to say.
- Record the time each step was taken.
This is why a fallback must be designed at build time. A model with no rollback, no rule and no off switch turns a bad hour into a bad week. Our general guide to the first hour of an incident covers communication in more depth.
Common causes, roughly in order of likelihood
| Cause | Typical sign | Where to look |
|---|---|---|
| Upstream data change | Sudden shift after a data load or source system update | Input logs, schema changes, recent changes to source systems |
| Feature pipeline bug | Predictions wrong after a code release | Recent deploys, training versus live feature values |
| Bad model release | Problem starts exactly when a new version went live | Model registry, release notes, evaluation report |
| Missing or default values | Many predictions identical or clustered | Null rates, default fill logic |
| Genuine sudden change in the world | Gradual or event-linked shift, inputs look valid | Business events, market changes |
| Infrastructure problem | Timeouts causing fallback values | Service logs, latency, memory |
In our experience the top two rows account for most real incidents. The model itself is rarely the thing that broke; something feeding it changed.
Finding which decisions were affected
Once contained, work out the blast radius. This is where logging decides whether it takes an hour or a fortnight.
- Use the prediction log to list every prediction since the suspected start time
- Check when inputs or predictions first diverged from normal, which often moves the start time earlier than assumed
- Rescore affected records with the previous model or corrected data
- Compare old and new predictions to find the decisions that would have been different
- Group them by consequence: already acted on, pending, or harmless
Without a prediction log with model versions and inputs, you are left reconstructing what happened from downstream systems, which is slow and incomplete. It is the strongest argument there is for logging, and we make it in model versioning and rollback too.
Repairing the damage
Rolling back stops new bad decisions. The ones already made need a business response, and that is not an engineering decision alone.
- Cancel or correct pending actions, such as unsent offers or unprocessed approvals
- Decide with the business owner how to handle actions already taken, such as honouring a discount or contacting affected customers
- Check whether any decisions affected people in ways with legal or regulatory implications
- Keep a list of affected records and what was done about each
The incident is not over when the model is fixed. It is over when every wrong decision it made has been found and dealt with.
The review afterwards
Hold a short, blame-free review within a week. Keep it practical.
- What happened, with a timeline
- How it was detected, and how it could have been detected sooner
- Why the existing checks did not catch it
- What was affected and how it was resolved
- Which new checks, alerts or tests will prevent a repeat, with owners and dates
The most useful outcome is nearly always a new automated check: a data test on the field that changed, an alert on the prediction distribution, or a behavioural test. Each incident should make the next one shorter.
Preparing before anything goes wrong
A one-page runbook per model covers most of it: how to roll back, how to switch to the fallback, who can authorise that, where the prediction log lives, which queries find affected records, and who to tell. Practise the rollback once.
SpiderHunts writes this runbook as part of every model we put into production through our machine learning service, and we support clients through incidents when they happen. The hope is always that the runbook gathers dust. Occasionally it earns its keep in a single morning.
Frequently asked questions
What should I do first when a machine learning model makes bad predictions?
What usually causes a model to suddenly go wrong?
How do we find which predictions were affected?
What is a model runbook?
Dealing with a model that has gone wrong?
Tell us what the model does and what people are seeing. We will help you contain it first and find the cause second, and we will be straight about how long that takes.