Model Drift: Why Your Model Gets Worse Over Time

Your model did not break. The world moved.

A lead-scoring model built in spring ranks enquiries beautifully for a few months. By winter the sales team has stopped looking at the scores. Nobody changed the code. What changed was the enquiries: a new ad campaign brought in smaller businesses, a competitor raised prices, and the questions people asked on the form shifted.

That slow decay is model drift. It is the normal condition of a machine learning model, not a sign that something went wrong in the build. A model is a snapshot of patterns in historical data, and history keeps getting longer. Customer-behaviour models, like a churn prediction model, tend to feel it first.

The two kinds of drift worth separating

People use the word loosely, but there are two different problems and they have different fixes.

Data drift is easier to catch because you can see it without waiting for outcomes. If the model was trained on orders averaging 80 pounds and live orders now average 130, you know something has shifted today.

Concept drift is harder. The inputs can look exactly the same while their meaning changes. You usually only find it by comparing predictions to what actually happened, which may take weeks or months to arrive.

Common causes of model drift

• Seasonality the training data did not cover. A model trained on six months of data has never seen December.
• Business changes you made yourself. New pricing, a new product line, a new market, a changed returns policy.
• Market shifts. Competitors, interest rates, supply problems, a change in what customers expect.
• The model changing behaviour. A fraud model blocks certain patterns, fraudsters adapt, and the patterns it learned disappear. Recommendation models do a gentler version of this.
• Upstream data changes that look like drift. A renamed category or a new default value in a form. Strictly a bug rather than drift, but it presents the same way.

That last one deserves emphasis. In our experience a good share of apparent drift turns out to be a data pipeline fault. Check the plumbing before you blame the world.

How to detect model drift

Detection comes in two layers, and you want both.

The first layer watches inputs. For each important feature, compare its recent distribution with the training distribution: averages, ranges, the share of each category, the share of missing values. Statistical tests such as population stability index or Kolmogorov-Smirnov give you a number to alert on, but a simple chart of weekly averages catches most real problems.

The second layer watches outcomes. Log every prediction, join it to the real result once that is known, and track accuracy by week. This is the only direct measure of whether the model still works. Our post on monitoring machine learning models in production covers the set-up in more detail.

Input monitoring tells you something changed. Outcome monitoring tells you whether it mattered. A model can survive a lot of input drift and still be accurate.

What to do when you find it

1. Rule out a data fault first. Look for renamed fields, new category values, unit changes and broken joins.
2. Measure how much accuracy has actually dropped, on recent labelled data rather than a feeling.
3. If the drop is small and the business impact low, note it and keep watching.
4. If it matters, retrain on a window that includes recent data and compare the new model against the current one on the same holdout period.
5. If retraining does not recover accuracy, suspect concept drift and revisit the features.
6. Deploy the replacement with a rollback ready, and record why it was changed.

What we would avoid is retraining automatically every time an input alarm fires. That replaces a slow decline with erratic behaviour, and it can bake a data fault straight into the new model.

When drift is a sign the model is the wrong tool

Some problems drift so fast that a model is always out of date. Pricing in a volatile commodity market, or demand for fashion items driven by social media, can move faster than any sensible retraining cycle.

In those cases a simpler approach that reacts to recent data directly, such as a moving average or a rule set someone updates weekly, can beat a sophisticated model that is always a quarter behind. It is less satisfying and often more accurate. SpiderHunts has recommended exactly that more than once when a client expected us to build something cleverer.

Building drift handling in from the start

Drift is cheap to handle when it is planned for and expensive when it is discovered. At build time, we would save the training data profile alongside the model, log every prediction with its inputs and model version, and agree how outcomes will be collected. Those three things cost very little upfront.

If you are about to commission a model, ask how drift will be detected before asking how accurate it is. Our machine learning service treats that as part of the scope rather than an optional extra.

Frequently asked questions