Shadow Deployments and A/B Tests for Machine Learning Models

Offline scores are a promise, not proof

A new model beats the old one on historical data by a comfortable margin. The temptation is to swap it in on Monday. Sometimes that works. Often the new model meets live data that is messier than the test set, a feature is calculated slightly differently in production, or its better accuracy turns out not to change anything the business cares about.

Shadow deployments and A/B tests are two ways to find out before committing. They answer different questions, and mixing them up is how teams end up with a month-long experiment that proves nothing.

Shadow mode versus A/B test

Shadow mode is about engineering confidence. An A/B test is about business evidence. You can do the first without the second; doing the second without the first is asking for trouble.

How to run a shadow deployment

1. Send the same live inputs to both models, ideally from the same code path the real system uses.
2. Log both predictions with a shared ID, timestamps and model versions.
3. Check the boring things first: latency, memory, errors, missing features.
4. Compare the predictions. Where do they agree, where do they differ, and are the differences concentrated in one segment?
5. Pull a sample of the biggest disagreements and have someone who knows the business look at them.
6. As outcomes arrive, score both models on the same live cases.

Step five is where shadow mode earns its keep. A pricing model that disagrees with the current one mostly on bulk trade orders is telling you something a single accuracy number never would.

One limitation to accept: shadow mode cannot measure effects that depend on acting on the prediction. If a recommendation model's value is in what customers click, a shadow model never gets clicked.

When an A/B test is worth running

A/B tests are expensive in time and attention. They are worth it when the business effect is genuinely uncertain and the volume exists to measure it.

• Recommendation, ranking and search models, where the effect depends on customer behaviour
• Pricing and discount models, where revenue rather than accuracy is the target
• Fraud or risk models, where changing thresholds trades losses against friction
• Any change that a senior person will want to see proven in money rather than in accuracy points

They are usually not worth it for back-office models with few cases, like forecasting monthly stock for 40 suppliers. There is not enough volume for a clean result, and a shadow comparison plus careful review gives you most of the confidence. The same small-volume problem applies to website tests, which we covered in A/B testing for small traffic sites.

Traps in A/B testing machine learning models

• Stopping early. Checking every day and ending the test the moment it looks good produces false wins. Decide the duration and sample size in advance.
• Assigning by the wrong unit. Split by customer, not by request, or one customer sees both models and contaminates both groups.
• Measuring the model's metric instead of the business one. Higher click-through with lower basket value is not a win.
• Ignoring label delay. A churn model test that ends after two weeks has measured nothing about churn.
• Feedback effects. A model that chooses what customers see also shapes the data future models train on.

A test that was going to be declared a success regardless of the numbers is not a test. Write down what result would make you keep the old model before you start.

A middle path: canary rollouts

Between shadow and full A/B testing sits the canary rollout. The new model takes a small share of real traffic, say five per cent, while you watch error rates, complaint volume and obvious business signals. If nothing looks wrong you widen it in steps.

A canary is about limiting damage rather than proving benefit. It is a sensible default for models where a full experiment is overkill but a straight swap feels reckless. It pairs naturally with good model versioning and rollback, because the whole point is being able to step back quickly.

How we usually sequence it

For most client models SpiderHunts runs a shadow period of two to three weeks, reviews disagreements with the people who use the predictions, then does a canary rollout. A formal A/B test is reserved for models where revenue impact is the question and the traffic can answer it.

If you are planning a model replacement and want help designing the comparison, our machine learning team can set up the logging and reports so the decision rests on evidence rather than on who argues hardest.

Frequently asked questions