How Accurate Does a Machine Learning Model Need to Be?

Ninety per cent of what

Somebody in a meeting will eventually ask what accuracy the model gets. The supplier says 92%, everyone nods, and a decision is made on a number that on its own means very little.

Accuracy is simply the share of predictions that turned out right. Whether 92% is excellent or useless depends on three things nobody mentioned in that meeting: how often the thing you are predicting happens, what each kind of mistake costs, and how good the current approach already is.

Why a high score can be worthless

Take an online retailer where 2% of orders are fraudulent. A model that marks every single order as genuine is 98% accurate. It also catches no fraud whatsoever. Nobody would buy it, yet on the headline figure it beats a genuinely useful model scoring 95%.

This is why accuracy is the wrong measure whenever one outcome is rare, which in business it nearly always is: fraud, churn, defects, missed appointments, bad debts. For those problems you want to know how many of the real cases the model finds, and how many false alarms it raises along the way. Those measures are called recall and precision, and we explain the trade-off between them here.

A model that predicts nothing ever happens is right most of the time about most businesses. That is not the same as being useful.

Start from the cost of being wrong

Every prediction can go wrong in two directions, and the two directions almost never cost the same. Put a price on each before discussing any target.

Once the costs are on paper, the question changes from 'how accurate' to 'at what point does the model save more than it costs'. That is a question a finance director can answer, which is the point.

Compare against what you do today

The honest benchmark is never perfection. It is the current process. If an experienced planner forecasts weekly demand to within 20% most weeks, a model that gets within 15% is an improvement worth having, even though it sounds unimpressive written down.

We always build a baseline first, and it is usually embarrassingly simple:

• Next week's sales will match the same week last year
• Customers who complained twice will cancel
• Jobs take as long as the average for that job type
• Every email goes to the team it went to last time for that sender

If the model cannot clearly beat the baseline, the project stops or changes shape. It happens more often than people expect, and finding out in week two is much cheaper than finding out after launch. Occasionally the baseline itself becomes the product, which is fine. A rule sometimes beats a model, and there is no shame in shipping the rule.

Human in the loop changes the target

Where a person reviews every prediction, the model only needs to make their job faster. Suggesting the right expense category 80% of the time means the finance assistant confirms most lines with one click and corrects the rest. That can halve the time spent, at an accuracy that would be unacceptable if the system posted to the ledger unattended.

Where the model acts alone, the bar rises sharply, and the sensible design is to act only on confident predictions and send the uncertain ones to a person. That way the automated portion can run at very high accuracy while the model still handles a large share of the volume. Our write-up on spend classification shows that pattern in practice.

A simple way to set the number

1. Write down the decision the model will change
2. Price a false alarm and a miss, even roughly
3. Measure the current process on the same past data
4. Estimate the value at a few accuracy levels, say the baseline, the baseline plus five points and plus ten
5. Decide whether a human reviews the output, and set separate targets for automated and reviewed cases
6. Agree the target, and the measure, before the model is built

Step six matters because accuracy figures are easy to flatter after the fact: test on an easy period, drop awkward cases, choose the kindest metric. Agreeing the test up front removes the temptation for everyone, suppliers included.

Real accuracy is lower than test accuracy

Expect the model to perform somewhat worse live than in testing, and worse still over time. Test data is cleaner than real data, and the world drifts. A sensible plan assumes a few points of slippage and has monitoring in place to catch more.

When SpiderHunts scopes a machine learning build, the accuracy target in the proposal is always expressed as money or hours against the current baseline, measured on a recent period the model never saw. It makes for less exciting proposals. It also makes it very clear, very early, whether the thing is worth paying for.

Frequently asked questions