Model Versioning and Rollback for Business-Critical Predictions
Last updated:
The question you will eventually be asked
A customer complains that they were refused a trade credit limit. Your finance director wants to know why. The model that made that decision was replaced two weeks ago. Can you say which version scored them, what inputs it saw and what it would have said under the current model?
For most businesses running a model without proper versioning the answer is no, or yes after two days of digging. Versioning is the practice that turns that into a five-minute query. Rollback is its partner: when a new model misbehaves, you go back to the last good one before the damage spreads.
What actually needs a version
Versioning code with Git is the easy part and most teams already do it. A model is the product of more than code.
| Artefact | Why it matters | Common way to version it |
|---|---|---|
| Training code | Defines how features and model are built | Git commit hash |
| Training data | Same code on different data gives a different model | Dated snapshot, or a query plus a fixed cut-off date |
| Settings | Hyperparameters, thresholds, feature lists | Config file committed with the code |
| Library versions | Upgrades can change results silently | Pinned requirements or a container image |
| Trained model file | The thing that actually makes predictions | Stored in a registry or bucket with a unique ID |
| Evaluation results | Evidence it was good enough to release | Report stored with the model ID |
Tie them together with one model version ID, and write that ID next to every prediction. That single habit answers most audit questions you will ever face.
A simple model registry is enough to start
A model registry is a list of trained models with their metadata and a status: candidate, live, retired. Tools like MLflow provide one, and the cloud ML platforms include their own. For a single model, a database table and a storage bucket with a clear naming scheme do the same job.
- Model ID, created date and who trained it
- Git commit, data snapshot reference and config
- Evaluation scores, including key segments
- Status and the date it went live or was retired
- A short note on why it was promoted
The note is underrated. Six months later, knowing that version 14 went live because version 13 over-predicted for new customers is more useful than any metric.
Designing for rollback
A rollback plan that has never been tested is a hope. The goal is that restoring the previous model is a boring configuration change, not an emergency engineering task.
- Keep at least the previous live model loaded or ready to load. An archive somewhere does not count.
- Select the live model through configuration, so switching is one change and no redeploy of code.
- Make sure the feature pipeline can still produce what the old model expects. This is the step that breaks rollbacks.
- Decide in advance what conditions trigger a rollback and who can authorise it.
- Practise it once, in production hours, when nothing is wrong.
Step three deserves a warning. If the new model added a feature and the pipeline was changed to drop an old one, the previous model cannot run any more. Keep pipelines backward compatible for at least one release, or version them alongside the model.
The time to find out your rollback does not work is a quiet Wednesday, not the morning after a bad release.
Rollback is not always enough
Rolling back the model stops new bad predictions. It does not undo the ones already made. If a pricing model issued wrong quotes for a day, or a risk model approved accounts it should have held, someone needs to find and deal with those decisions.
This is where prediction logging with version IDs pays off. You can list every decision made by the faulty version, rescore them with the previous model and see which would have differed. We cover the wider response in machine learning incident response, and there is related thinking for language model features in AI integration rollback and safety.
How much versioning is appropriate
Match the discipline to the consequence of a bad prediction.
- Internal forecast used as one input to planning: Git, saved model files with dates, and a note of the data cut-off
- Customer-facing recommendations: add a registry, prediction logging and a config-based switch
- Pricing, credit, fraud or anything affecting people: full lineage from data to prediction, tested rollback, approval records and retention in line with your legal obligations
Over-engineering is a real risk for small teams. A registry product for one monthly forecast is ceremony. A pricing engine with no way to say which model quoted a price is negligence.
Where we start with clients
When SpiderHunts inherits a model through our machine learning work, the first check is whether the current live model can be reproduced from its code and data. Surprisingly often it cannot; the training data was a CSV on someone's laptop. Fixing that comes before any improvement, because improving a model you cannot reproduce means you cannot go back.
If you are designing the first deployment rather than rescuing one, our machine learning deployment guide covers the wider set-up that versioning slots into.
Frequently asked questions
What is model versioning?
Do we need a model registry?
How fast should a model rollback be?
Should we version training data too?
Could you roll your model back today?
If the honest answer is not sure, tell us how your model is deployed. We will point out what is missing for a safe rollback and how much work it is to add.