Why normal CI is insufficient
Continuous integration tests that code does what it should. A retrained model can pass every unit test and behave quite differently, because the change is in learned parameters rather than logic.
The pipeline therefore needs gates on data and behaviour as well as code. Without them, a retraining job can quietly push a worse model into production on schedule.
The gates worth having
- Code tests - the ordinary ones, including feature transformation logic, which is where subtle bugs live.
- Data validation - schema, ranges, missing-value rates and row counts on the training set, with the job failing rather than warning.
- Training success - convergence and sane parameters.
- Performance gate - the new model must beat the current one on a fixed held-out set by an agreed margin.
- Behavioural checks - a set of known cases with expected outcomes, including edge cases and any case that previously caused an incident.
- Subgroup check - no important group materially worse than before.
The behavioural set is the most valuable and least common. Each production surprise should add a case to it, so the same mistake cannot ship twice.
A performance gate that is not gameable
The comparison set must be fixed and separate from anything used in training or tuning. If it moves with each release, the gate compares two different things and means little.
Require a margin rather than any improvement, so noise alone cannot pass a model. And decide in advance what happens on failure - block, or alert and hold for review - because that is a policy question, not a technical one.
Deploy gradually
| Stage | What it catches |
|---|---|
| Shadow - score without acting | Pipeline errors, wildly different outputs |
| Small share of live traffic | Real-world behaviour on a limited blast radius |
| Progressive increase | Effects that only appear at volume |
| Full | - |
Keep the previous model deployable throughout, and make rollback a single documented action rather than a rebuild. Most teams discover their rollback is not actually one step at the worst possible moment.
Version everything together
A deployment is the combination of model, feature code and configuration. Versioning them separately allows mismatches - a new model with old feature code produces silently wrong predictions rather than an error.
Release them as one unit with one version number. It is the single change that prevents the most confusing class of production failure. Our note on versioning and rollback covers the mechanics.
A retrained model is a behaviour change. It deserves more scrutiny than a code change, not less.