Data Leakage: The Silent Bug That Makes Models Look Brilliant
Last updated:
The result that should worry you
A supplier sends over a churn model with 97% accuracy. The board is delighted. Three months after launch, it is barely better than guessing. Nothing in the code is broken, nothing crashed, and no error was logged.
This is the classic signature of data leakage. The model learned from information that was available in the historical data but will never be available when a real prediction is needed. In testing it was effectively allowed to peek at the answer.
Leakage is the most common serious bug we find when reviewing other people's models, and it is almost never deliberate. It creeps in because historical data is a snapshot of everything that eventually happened, and the model cannot tell the difference between cause and aftermath.
A worked example
Take the churn model above. The training data is one row per customer, with a column saying whether they cancelled. Among the inputs is 'number of calls to the retention team' and 'account status notes'.
Customers who cancelled were, almost by definition, passed to the retention team, and their notes say things like 'requested cancellation'. The model learns that retention calls predict churn. Perfectly true, and perfectly useless: by the time a retention call happens, the customer has already decided to leave. At the moment you would want to intervene, that feature is zero for everyone.
For every input, ask one question: on the day we would make this prediction, would we actually know this value?
The common forms of leakage
- Target leakage. An input that is a consequence of the outcome, like the retention calls above, or a 'write-off reason' in a late-payment model.
- Time leakage. Using values calculated with future data, such as a customer's total annual spend when predicting something in March.
- Duplicate leakage. Near-identical records, such as the same invoice exported twice, landing in both training and test sets.
- Group leakage. Several records from the same customer, patient or machine split across training and test, so the model recognises the individual rather than the pattern.
- Preprocessing leakage. Scaling, filling missing values or selecting features using statistics from the whole dataset, including the test portion.
- Overwritten fields. CRM or ERP fields that hold today's value rather than the value at the time, like a customer segment updated after they churned.
How to spot it
- Distrust surprising accuracy. If a business prediction scores far better than experienced staff manage, investigate before celebrating.
- Look at feature importance. One input dominating everything is a warning sign, especially one recorded late in a process.
- Walk through a single real case. Pick one customer, go to the prediction date, and check which inputs actually existed then.
- Check when each field is populated. Ask the people who use the system, not just the data dictionary.
- Test on a later period. Train on older data and test on the most recent months. A large drop compared with random splitting often points to leakage.
Fixes for each type
| Leak | Typical symptom | Fix |
|---|---|---|
| Consequence of the outcome used as input | One dominant feature, near-perfect scores | Remove it, or use only its value before the prediction date |
| Future values in aggregates | Great backtest, poor live results | Calculate features as at the prediction date |
| Same entity in train and test | Performance drops sharply for new customers | Split by customer, site or machine |
| Preprocessing on full dataset | Small but consistent optimism | Fit preprocessing on training data only, inside a pipeline |
| Overwritten system fields | Model fails after launch without clear cause | Use change history or build snapshots going forward |
The overwritten-field problem is the hardest, because the historical truth may simply be gone. When that happens we either find an audit log, or start saving dated snapshots and accept that a proper model has to wait for enough history to build up.
Building leakage out of the process
Most leakage is prevented by structure rather than vigilance. Build training data with point-in-time joins, attaching to each historical event only the values that existed at that timestamp. Keep all preprocessing inside a single pipeline object fitted on training data. Split by time and by entity as the default, and only use random splits when there is a good reason.
Keeping feature calculations in one shared place, whether a simple set of tables or a full feature store, makes point-in-time correctness far easier. Leakage also appears in forecasting when promotions or stock levels are known only in hindsight, which we touch on in forecasting demand with your own data.
A second pair of eyes is cheap
Leakage is easy to miss when you built the model yourself, because you know what every field is supposed to mean. At SpiderHunts, every model gets a review from someone who did not build it, with the explicit job of asking 'would we have known this?' for every input. It takes a few hours and has saved more than one project from a painful launch.
If you have a model that performs suspiciously well, or one that fell apart after going live, our data science team can review it. Usually the fix is straightforward once the leak is found. The realistic accuracy is lower, but it is accuracy you will actually get.
Frequently asked questions
What is data leakage in machine learning?
How do I know if my model has data leakage?
What is target leakage?
Should I use a random or time-based train-test split?
Can data leakage be fixed after a model is built?
Model results that look too good to be true?
They may be. Share how the model was built and tested, and we will review it for leakage and tell you what accuracy you can realistically expect in production.