Testing Machine Learning Systems Before Customers See Them

A good test score is where testing starts

The model scored well on the holdout set, so it shipped. Two days later a customer in Northern Ireland was quoted a delivery date three weeks out, because postcodes starting with BT had barely appeared in training data and the model extrapolated badly. The overall accuracy was fine. The failure was specific, visible and embarrassing.

Machine learning systems need testing in more dimensions than ordinary software, because they can be wrong in ways that an average score hides. Engineers who test web apps carefully sometimes skip this entirely for models, as if the evaluation metric were the test suite. It is not.

The layers of testing for a machine learning system

Most of these are ordinary engineering practice applied to a new kind of component. If your team writes tests for other software, none of this should feel exotic.

Data tests come first

More model failures start in the data than in the model. Test incoming data the same way you would validate a form submission.

• Every expected column is present, with the right type
• Numeric values fall in plausible ranges; a negative order quantity should stop the job
• Category fields contain only known values, or new ones are flagged
• Missing values stay within historical levels
• Row counts are in a sensible band compared with recent runs
• No duplicate keys where there should be none

Tools such as Great Expectations or pandera can help, but a set of plain assertions in the pipeline gets you most of the way. The important decision is what happens on failure: stop, warn, or quarantine the bad rows.

Behavioural tests catch what averages hide

Behavioural testing means writing down how the model should behave on particular inputs and checking it does. It borrows from ordinary unit testing and is the most underused technique in business machine learning.

1. Known cases. A handful of real examples where the right answer is obvious. A long-standing customer who always pays on time should not be flagged high risk.
2. Invariance tests. Changing something irrelevant should not change the prediction much, such as the capitalisation of a company name.
3. Directional tests. Increasing something should move the prediction the expected way. More late payments should not lower a risk score.
4. Edge cases. New customers, zero values, very large orders, rare regions.
5. Segment floors. Accuracy for each important group stays above a minimum, whatever the average says.

Ask the people who use the predictions to supply the known cases. They know which mistakes would be embarrassing far better than a data scientist does.

Quality gates for releasing a model

A quality gate is a written, automated rule that a candidate model must pass before it can go live. It turns a judgement call made under deadline pressure into a check agreed calmly in advance.

• Beats or matches the current model on the agreed business metric, on the same recent holdout period
• No important segment worse by more than an agreed margin
• All behavioural tests pass
• Prediction latency and memory within limits
• Explanations or documentation generated where required

Agree the release criteria before you see the results. Otherwise the criteria somehow end up being whatever the new model achieved.

The same principle drives how we test language model features, covered in evaluating AI output quality.

The training and serving mismatch

One class of bug deserves its own test: the model sees different inputs in production than it did in training. A feature computed in SQL for training and reimplemented in application code for serving will drift apart, usually around dates, rounding, time zones or missing values.

The test is simple. Take a sample of records, compute features and predictions through the offline pipeline and through the live service, and assert they match within a tiny tolerance. Run it on every release. It has caught more real bugs in our projects than any clever evaluation metric.

Testing does not end at release

Some failures only appear with live data, real users and real volume. Shadow deployment, a gradual rollout and monitoring are the final layers of testing, not a separate activity. We describe how to run them in shadow deployments and A/B tests.

When SpiderHunts builds a model as part of a custom software or machine learning project, the test suite is delivered with it and runs in the same pipeline as the rest of the code. A model without tests is a model the next developer will be frightened to touch.

Frequently asked questions