How We Support Models After Launch

Models decay even when nothing breaks

Ordinary software keeps doing what it did on launch day until something changes in the code. A machine learning model does not. It learned from a picture of your business at a moment in time, and the business keeps moving. Customers change how they buy, a new product line arrives, a price rise shifts behaviour, a supplier changes a data format. Nothing crashes. The predictions just slowly become less right.

That is why launching a model is the start of its useful life, not the end of the project. How we support a model after launch is designed around catching that decay early, deciding what to do about it and keeping a record of every change.

The first 90 days: the warranty

Every SpiderHunts build carries a 90-day warranty from go-live. For a machine learning system the boundary needs stating carefully, because “the model got something wrong” is not automatically a defect. Models make mistakes by design; the question is whether the system behaves as the signed specification says.

The general principles of our warranty are the same as for any project; if we misunderstood you, that is our cost, and if the requirement changed, it is a change request.

What we monitor, and why each matters

1. Pipeline health. Did the data arrive, did the job run, were predictions written? The boring failures are the most common.
2. Input drift. Is the live data still shaped like the training data? A sudden jump in average order size or a new category appearing is an early warning.
3. Prediction drift. Has the share of high-risk scores changed sharply without an obvious business reason?
4. Real accuracy. As outcomes arrive, how right were the predictions made weeks ago?
5. User feedback. How often are people marking predictions as wrong, and is there a pattern?
6. Latency and cost. Is it still fast enough and cheap enough at current volume?

Each has a threshold agreed at go-live and an alert that reaches a named person. We look at the MLOps tooling for mid-sized companies separately; the point here is that monitoring is designed in from the start, not added when someone complains.

Retraining without surprises

Retraining is the routine way a model keeps up. We version the data, the code and the model together, so any retrain can be repeated and any past version restored. A retrain is triggered either on a schedule suited to how fast your business changes, or by a monitoring threshold being crossed.

• The new model is trained on the refreshed data using the same, versioned pipeline
• It is scored on the same evaluation approach as the live model, on the most recent data
• It replaces the live model only if it does at least as well, including on key segments
• For higher-stakes models, a person on your side approves the switch
• The old version stays available for instant rollback

A retrain that nobody checked is not maintenance. It is a new model deployed without a go-live decision.

When a retrain is not enough

Sometimes monitoring reveals something retraining cannot fix. The business process changed, so the outcome the model predicts no longer means the same thing. A key data source was switched off. A new regulation means a factor the model used can no longer be used. In those cases we stop, explain what changed and propose options, which can include switching the model off and returning to the fallback rule while a revised version is built.

We would rather tell you a model has stopped earning its place than keep it running out of habit. That honesty is part of the support, and it is also why every model we deploy has a fallback.

The retainer, and the alternatives

After the warranty, most clients with a model in production keep an optional monthly retainer covering monitoring, scheduled retraining, investigation of alerts and a small block of hours for improvements. It can be cancelled monthly. As a budgeting guide, allow roughly 15–20% of the original build cost a year for evaluation and maintenance, whoever does it.

You do not have to use us. The handover package includes the pipeline, the retraining procedure, the evaluation set and the monitoring configuration, so your own team or another supplier can take over, as described in how we hand a project over. What we advise against is having nobody watching at all. That is how a useful model becomes a liability without anyone noticing, and it is the gap our machine learning service is designed to close.

Frequently asked questions