Imbalanced Data: Why Rare Events Are Hard to Predict
Last updated:
The 99.8% accurate model that catches nothing
Picture an online retailer processing 20,000 orders a month, of which roughly 40 turn out to be fraudulent. A model that simply says 'not fraud' for every order is 99.8% accurate. It is also useless.
This is the imbalanced data problem, and it affects most of the predictions businesses care about: fraud, equipment failure, defective parts, customers about to leave, claims that end in litigation. The event you want to catch is rare, which is exactly why it is valuable to catch and exactly why a naive model ignores it.
Why rare events defeat ordinary models
Most algorithms try to minimise total errors. When one outcome makes up 99% of the data, the cheapest way to do that is to lean heavily towards it. The rare cases barely register in the maths.
There is a second, less discussed problem. Forty fraud cases a month is 480 a year, and those 480 may involve a dozen different fraud techniques. The model is not learning one pattern from 480 examples, it is learning twelve patterns from around 40 each. That is small data wearing a big-data disguise.
Measure the right thing
Accuracy should be banned from rare-event conversations. These are the measures that tell you what you need to know.
| Metric | Question it answers | Why it matters |
|---|---|---|
| Recall | Of all the real events, how many did we catch? | Missing fraud or failures is usually the expensive error |
| Precision | Of everything we flagged, how much was real? | Low precision buries your team in false alarms |
| Precision-recall curve | How does the trade-off shift as the threshold moves? | Shows the full range of options, not one setting |
| Alerts per day at a given recall | How much work does this create? | Translates statistics into staffing |
In practice we find the last row the most useful when talking to operations managers. 'This setting catches about seven in ten fraud cases and produces around 25 alerts a day to review' is a sentence people can make a decision about.
The threshold is a business decision
A model gives each case a score. Where you draw the line between 'flag' and 'ignore' is not a data science question. It depends on costs.
An illustrative example: suppose a missed fraudulent order costs £300 on average, and reviewing a flagged order takes a staff member five minutes, costing about £2. At that ratio you can afford to review a lot of false alarms to catch one more real case, so the threshold should be set low. If instead each false alarm meant blocking a genuine customer's order and losing them, the calculation changes completely.
- Estimate the cost of a missed event
- Estimate the cost of a false alarm, including customer friction
- Work out how many alerts your team can realistically handle each day
- Pick the threshold that minimises total cost within that capacity
- Revisit it every few months as volumes and fraud patterns change
What actually helps
- Class weights. Telling the algorithm that errors on the rare class matter more. Simple, and usually the first thing to try.
- Resampling. Over-sampling rare cases or under-sampling common ones, including techniques such as SMOTE. It can help a little, but it does not create information that was not there, and it distorts probability scores unless you correct for it.
- Better features. Almost always the biggest gain. For fraud, things like mismatch between billing and delivery country or account age at first order.
- More positive examples. Pooling several years of history, or recording near-misses and suspected cases, not just confirmed ones.
- Splitting the problem. Separate models or rules for distinct types of event can outperform one model trying to learn them all.
We are sceptical of projects where resampling is presented as the main solution. It tends to improve a chart in a notebook more than it improves results in production.
When anomaly detection is the better tool
If you have very few confirmed examples, or the rare events keep changing shape, a supervised model may not be the right approach at all. Anomaly detection learns what normal looks like and flags what does not fit, without needing many labelled bad cases.
It produces more false alarms and cannot tell you what kind of problem it found, but it catches things nobody has seen before. We compare the approaches in anomaly detection for business operations. For a sector-specific view of the fraud case, see AI fraud detection in financial services.
How few events is too few?
There is no fixed floor, but as a rough guide, fewer than about 50 confirmed examples of the event makes a trustworthy supervised model very difficult, and a few hundred is a much more comfortable place to start. Below that, rules written with experts plus anomaly detection is often the honest recommendation.
When SpiderHunts takes on a rare-event project, the first deliverable is usually a precision-recall analysis in business terms: how many alerts, how many catches, what it would cost to run. Sometimes that shows a model is worth it; sometimes it shows the existing manual checks are already close to the best achievable. Either way the decision is made with real numbers, which is what our machine learning work is built around.
Frequently asked questions
What is imbalanced data?
Why is accuracy misleading for imbalanced data?
Does SMOTE fix imbalanced data?
How do I choose a classification threshold?
Can machine learning predict very rare events?
Trying to predict something that rarely happens?
Tell us how often the event occurs and what a miss costs you. We will tell you whether a model can realistically catch it, and at what cost in false alarms.