How Often Should You Retrain a Machine Learning Model?

There is no universal number

Somebody always asks for a rule of thumb, so here is the honest version: most business models we see do fine retrained somewhere between monthly and quarterly. That range is wide because the question is really three questions stacked together.

How quickly does the thing you are predicting change? How long until you know whether a prediction was right? And how much does a retrain cost in compute, checking and risk? Answer those and the schedule more or less writes itself.

Three ways to decide when to retrain

In practice we combine them. A performance trigger does the real work, a drift alert prompts someone to take a look, and a calendar schedule makes sure no model goes a year untouched just because monitoring quietly stopped.

Label delay changes everything

Label delay is the gap between making a prediction and learning the true answer. It is the most overlooked factor in retraining decisions.

• A next-day delivery estimate has a label delay of about a day. You can measure accuracy weekly and retrain quickly.
• A 90-day churn prediction has a label delay of at least 90 days. By the time you know it was wrong, the world has moved again.
• A loan default model may wait a year or more for reliable labels.
• A product recommendation gets a rough label within minutes, in clicks, and a better one within weeks, in returns.

With long label delay, performance triggers are too slow on their own. You lean more on input drift, on proxy signals that arrive sooner, and on retraining with whatever labels have matured. It is less precise, and there is no way around that.

What a retrain should include

Rerunning the training script is the easy tenth of a retrain. A retrain that skips the checks is how a good model gets replaced by a worse one on a Friday afternoon.

1. Pull a fresh training window, and decide deliberately whether old data is kept, down-weighted or dropped.
2. Run the same data quality checks used at the original build.
3. Train the candidate model with the same code and pinned library versions.
4. Evaluate candidate and current model on the same recent holdout period.
5. Check performance on important segments as well as the overall score.
6. Promote only if the candidate is better, or at least no worse, and record the decision.
7. Keep the previous model ready to restore.

Step five catches a lot. A new model can gain half a point overall while getting noticeably worse for your largest customers or for one region, and the headline figure hides it.

How much history to train on

Retraining frequency and training window are linked. A model retrained monthly on only the last three months reacts quickly and forgets last Christmas. A model retrained on five years of data is stable and slow to notice change.

For seasonal businesses we usually keep at least one full year plus recent months, sometimes weighting recent data more heavily. For fast-moving behaviour, a shorter window can win. The way to choose is to backtest: pretend it is six different points in the past, retrain with each window, and see which would have predicted the following month best. It takes a day and removes a lot of guesswork.

Should retraining be automated?

Automating the training run is sensible. Automating the decision to put the result live usually is not, at least at first.

A fully automated retrain-and-deploy loop is fine for low-stakes models with plentiful labels, like ranking products on a category page. For anything touching money, credit, pricing or people, we keep a human approval step: the pipeline produces the candidate and a comparison report, and a person clicks promote. It takes minutes and has saved more than one model from a bad data week.

Automate the work of retraining. Keep a person on the decision to trust the result, until you have a year of evidence the automation gets it right.

When retraining will not help

If accuracy was never good, retraining more often will not fix it. The same goes for a model whose inputs no longer carry the signal, or where a key data source has disappeared. Those need a rebuild, a different approach, or an honest decision that a rule beats a model here.

Retraining also cannot rescue a model that is quietly being fed bad data. If you are seeing accuracy fall, read our note on why models drift and rule out pipeline faults before scheduling anything.

At SpiderHunts we set the retraining policy as part of the data science handover document: trigger thresholds, window, checks and who approves. It is a page long and prevents most of the arguments later.

Frequently asked questions