How We Measure Whether a Model Is Worth Deploying
Last updated:
Accuracy is the wrong headline number
A model that is 95% accurate sounds excellent until you learn that 95% of your invoices are paid on time. A model that predicts “on time” for every invoice scores 95% and is completely useless. This is the most common way machine learning results mislead business owners, and it is why we almost never lead a report with accuracy.
Instead we measure a model the way the business will feel it: how many of the cases it flags are real, how many real cases it misses, and what each of those errors costs in money and time. The go-live decision is made on those numbers.
Check one: does it beat how the decision is made today?
Every model is compared against a baseline, which is usually the rule of thumb your team already uses written as code. If the credit controller chases every account over 30 days late, that rule is the benchmark. The model has to do clearly better on the same cases, or it does not ship.
We agree the margin that counts as “clearly better” in writing before training, during the proof of value. Deciding the bar after seeing the results is how teams talk themselves into deploying models that add maintenance without adding value.
Check two: what does each kind of mistake cost?
Models make two kinds of error, and they almost never cost the same. We put a rough value on each with you, then choose the model's threshold to minimise the total cost rather than to maximise a score.
| Example use | Cost of a false alarm | Cost of a miss | Usual setting |
|---|---|---|---|
| Chasing late payers early | A polite call to a good customer | An unpaid invoice written off | Flag generously |
| Fraud check at checkout | A genuine customer blocked | A chargeback | Balance carefully; route middle cases to review |
| Predictive maintenance | An unnecessary inspection | An unplanned breakdown | Flag generously |
| Automatic discount offers | Margin given away needlessly | A customer who might have left | Flag conservatively |
This is a business conversation, not a technical one. The model is the same; where you set the threshold decides whether it makes or loses money.
Check three: tested on the future, not the past
We train on older history and test on the most recent months the model has never seen. Shuffling all the data together and testing on a random sample produces flattering results, because the model gets to learn from periods after the ones it is being tested on. Real deployment never offers that luxury.
We also hunt for leakage: any field that quietly contains the answer. A “payment reminder sent” flag that only exists for accounts that went late will make a late-payment model look superb in testing and useless in production. The domain expert on your side is usually the one who spots these, which is why they review predictions with us.
When a model looks too good, the first suspect is the data, not the algorithm.
Check four: does it work for the cases that matter most?
Overall numbers can hide a model that fails exactly where it matters. So we break results down by the segments you care about before recommending go-live.
- Your largest customers or highest-value orders, separately from the long tail
- Each region, branch, channel or product family
- New customers with little history compared with established ones
- Any group of people where uneven performance would be unfair or unlawful
If the model is strong overall and weak for one segment, the options are to fix it, exclude that segment from automated use, or send those cases to a person. What we do not do is ship it and hope nobody notices.
Check five: shadow mode against live cases
Before a model influences anything, it runs in shadow mode. It receives real live cases and makes predictions that are logged but not shown to anyone, while the business carries on as normal. After a few weeks we compare what the model would have done with what actually happened and what your team decided.
Shadow mode catches problems historical testing cannot: data that arrives late or in a different format in production, fields that are filled in differently in real time, and seasonal quirks. It costs a little calendar time and is the cheapest insurance a launch can buy.
Check six: the operational tests, and the go-live document
A model that is accurate but slow, expensive or fragile is not ready either. We check prediction speed against what the system needs, running cost at realistic volume, behaviour when an input is missing, and what happens if the model service is unavailable. There must always be a fallback, even if it is the old rule.
All six checks are summarised in a short go-live document you sign off, alongside the monitoring thresholds that would trigger a review after launch. This is the same discipline we apply to other AI features in how we evaluate whether an AI feature is good enough, and it is part of every machine learning build we deliver.
Frequently asked questions
What accuracy does a machine learning model need before deployment?
How long does shadow mode last?
Can a model be deployed for some cases but not others?
Who makes the final go-live decision?
Have a model nobody is sure about?
Send us what you have and how it was tested. We will tell you whether it is ready to deploy, and what we would check before trusting it.