The dependency nobody planned
A prediction service gets added to a checkout, an order flow or an approval process. It works, so it becomes load-bearing. Then it is unavailable - a deployment, a dependency, an expired certificate - and nobody has decided what the process should do.
The default is whatever the code happens to do, which is often to throw an error and stop. A recommendation failure taking down a checkout is a real and avoidable outcome.
Four options, chosen per use case
| Strategy | Behaviour | Suits |
|---|---|---|
| Fail open | Proceed as if approved or unflagged | Low-risk decisions where blocking costs more |
| Fail closed | Block or route to manual review | Risk controls where proceeding is dangerous |
| Cached value | Serve the last known prediction | Slowly-changing scores |
| Rule fallback | Apply a simple pre-agreed rule | Where a crude answer beats none |
The choice belongs to the business owner, not the engineer, because it is a risk decision. It also differs by use case within one organisation - a fraud check should fail closed while a product recommendation should quietly disappear.
The rule fallback is underrated
Keeping the rule-based logic the model replaced, rather than deleting it, gives a fallback that is already understood and already correct enough to have run the business previously.
It needs occasional exercise to stay working. A rule path untouched for two years will have drifted out of step with the data around it. Running it periodically, even just in shadow, keeps it honest.
Make degradation visible
When the fallback engages, people need to know. A silent switch to default recommendations looks like a model that suddenly got worse, and teams can spend days investigating a model that is not running.
- Log every fallback with its reason, and alert if the rate crosses a threshold
- Show a quiet indicator in internal tools when scores are cached or defaulted
- Record which path each decision took, so later analysis is not contaminated
- Report fallback rate alongside accuracy in routine monitoring
That third point matters for training data. Decisions made by the fallback should not be attributed to the model when you next evaluate it.
Test it deliberately
A fallback that has never been exercised is a plan rather than a mechanism. Turn the model off in a test environment - and periodically in production, during a quiet window - and confirm the business process continues.
This is unglamorous and it is the difference between a designed degradation and an incident. The failure everyone remembers is rarely the model being wrong; it is the model being absent and nothing having been decided.
An untested fallback is a note saying what you hoped would happen.