Software breaks loudly, models degrade quietly
When an application fails, someone notices immediately. When a model degrades, it keeps producing plausible numbers that are gradually less correct, and nobody notices for months.
That difference is why support for a machine learning system is not optional in the way support for a static website arguably is. Without it, the system's value declines invisibly.
What the arrangement should include
- Monitoring - input distributions, prediction distributions, accuracy where outcomes arrive, and failure rates
- Scheduled retraining - at an agreed cadence, with gates before anything is deployed
- Drift investigation - someone responsible for looking when an alert fires
- Dependency maintenance - library and security updates, which do not stop because the model is stable
- Incident response - defined response times when predictions are wrong or the service is down
- Periodic review - is this still worth running, and is it still solving the right problem
The last one is the most valuable and the least common. A model quietly running two years after the process it supported changed is a cost with no benefit.
A realistic budget
Ongoing cost is commonly a meaningful annual percentage of the original build, and the honest answer is that it varies with how fast your data changes and how critical the model is.
| Factor | Pushes cost up |
|---|---|
| Fast-changing data | More frequent retraining and monitoring |
| Critical decisions | Tighter response times, more oversight |
| Regulated context | Documentation, audit, fairness testing |
| Many models | Each one needs its own attention |
| Complex pipeline | More that can break upstream |
Whatever the figure, get it in the business case at the start. A project justified on build cost alone will look like it failed when the running cost appears in year two.
Questions worth asking a supplier
- What exactly do you monitor, and what triggers you to act?
- How often will the model be retrained, and who approves deployment?
- What is the response time if predictions are visibly wrong?
- What happens if you are unavailable - could we retrain this ourselves?
- How will you tell us the model is no longer worth running?
Question four matters more than it seems. If only the supplier can retrain, you have a dependency for as long as the model runs, and that should be a conscious choice rather than a discovery.
Support in-house instead
Bringing support in-house is reasonable and needs the handover to be genuine: code, documentation, environment, monitoring and someone trained. Our note on handing over a machine learning system covers what that involves.
The common failure is a handover that happens on paper while the knowledge stays with the supplier. Test it by having your team retrain and deploy once, with the supplier watching rather than doing.
Nobody notices a model getting worse. That is exactly why support is not optional.