Reading a Confusion Matrix Without a Data Science Degree
Last updated:
The most useful slide in the deck
When a data team presents a model, there is usually a slide with a two-by-two grid of numbers on it. People skim past it to the accuracy figure, which is a pity, because the grid tells you far more. It is called a confusion matrix, a name that does it no favours, and it takes about two minutes to learn to read.
Once you can read it you can ask the questions that actually decide whether a model helps, and you will be much harder to impress with a single percentage.
The four boxes
Take a model that predicts whether a customer at a van hire firm will return the vehicle late. It was tested on 1,000 past rentals where the outcome is known. Here is an illustrative result:
| Model predicted late | Model predicted on time | |
|---|---|---|
| Actually late | 90 (caught) | 30 (missed) |
| Actually on time | 60 (false alarm) | 820 (correctly left alone) |
Rows are what really happened. Columns are what the model said. Reading the boxes:
- Caught, 90: the model said late and they were late. In the jargon, true positives.
- Missed, 30: the model said on time but they were late. False negatives.
- False alarm, 60: the model said late but they were on time. False positives.
- Correctly left alone, 820: the model said on time and they were. True negatives.
The diagonal from top left to bottom right is where the model was right. Everything else is a mistake, split by type.
Every familiar metric comes from these four numbers
All the percentages you get quoted are just different ways of dividing up the same grid. Using the example:
| Metric | Question it answers | Calculation | Result |
|---|---|---|---|
| Accuracy | How often was the model right overall | (90 + 820) / 1,000 | 91% |
| Recall | Of the late returns, how many did it catch | 90 / (90 + 30) | 75% |
| Precision | Of those it flagged, how many were really late | 90 / (90 + 60) | 60% |
| False alarm rate | Of the on-time customers, how many were wrongly flagged | 60 / (60 + 820) | 7% |
Notice that 91% accuracy sounds excellent while 60% precision sounds mediocre, and they describe the same model. Neither is the truth on its own. The grid is. We cover the balance between the middle two in precision vs recall.
Turn the boxes into money
This is where a confusion matrix becomes a business case. Put a value on each box. For the van hire firm, as rough illustrations:
- A caught late return lets staff call ahead and rebook the next customer, saving perhaps 80 pounds in disruption
- A missed late return costs the full disruption, say 120 pounds, as today
- A false alarm costs a courtesy call and a slightly irritated customer, say 5 pounds
- A correct 'on time' costs nothing and saves nothing
Across 1,000 rentals: 90 caught at 80 pounds saved is 7,200. Sixty false alarms at 5 pounds is 300. The model's net value is around 6,900 pounds per thousand rentals, before running costs. If the firm does 20,000 rentals a year, that is a number worth discussing. If it does 2,000, perhaps not.
Accuracy is a statistic. A confusion matrix with costs attached is a business case.
Questions to ask when you are shown one
- Which period is the test data from, and did the model ever see it during training
- How does the grid compare with a simple rule, such as flagging anyone who was late before
- What threshold produced this grid, and what does it look like at a stricter or looser one
- Is the mix of outcomes in the test data similar to real life
- How does the grid look for different customer groups, regions or products
The fifth question is the one most often skipped. A model can perform well overall and badly for a particular segment, such as new customers or one region. If the model makes decisions about people, checking this by group is also part of treating them fairly, and in some settings part of your legal obligations.
When there are more than two outcomes
A model sorting emails into six departments has a six-by-six grid. It looks intimidating and reads the same way: rows are the true department, columns are the model's choice, the diagonal is correct. The interesting part is where the off-diagonal numbers cluster.
If most mistakes are billing emails sent to accounts, and vice versa, that tells you those two categories overlap in practice. Sometimes the fix is a better model. Quite often it is merging the categories or clarifying the definitions staff use, which is a process change, not a technical one.
Making it part of routine monitoring
A confusion matrix is not only a sign-off document. Recomputing it every month on recent predictions, once outcomes are known, is one of the simplest ways to spot a model going stale. If the missed box starts growing, something has changed.
In the dashboards SpiderHunts builds for machine learning systems, the confusion matrix sits next to its money translation, updated as outcomes arrive. Managers look at the money. The data team looks at the grid. Both are looking at the same truth. If you need a refresher on the bigger picture first, our business guide to machine learning is a good start.
Frequently asked questions
Why is it called a confusion matrix?
What is a false positive versus a false negative?
Can a confusion matrix be used for regression models?
How often should a confusion matrix be recalculated?
Been handed a confusion matrix and asked to sign off?
Send it over with a note on what the model decides. We will translate the four numbers into business terms and tell you what questions to ask before approving.