Overfitting Explained: Why a Model That Aces Testing Can Fail in Real Life
Last updated:
The student who memorised the answers
Picture a student given last year's exam paper with the answers, a week before their own exam. One student learns the method behind each answer. Another memorises the answers themselves. On last year's paper both score full marks. On this year's paper, only one of them passes.
A model that memorises is overfitting. It has scored perfectly on the examples it was shown, and it has learned very little that transfers to new ones. The dangerous part is that, if you only ever check it against the paper it studied, it looks like the star pupil.
What it looks like in a business
A regional chain of opticians builds a model to predict which customers will book an eye test when reminded. On the historical data it is 97% accurate. In the first month live, its predictions are barely better than a guess.
When the team looks closer, the model has latched onto details that happened to line up in the past: a particular customer ID range that coincided with one busy store, a reminder template used for a few months, a handful of staff names. None of those patterns are real in any useful sense. They were coincidences in a limited dataset, and the model had enough flexibility to memorise them.
An overfitted model has not learned your business. It has learned your spreadsheet.
Why it happens
- The model is too flexible for the data. A very complex model given a few thousand rows can carve out a rule for almost every row.
- Too many inputs. Hundreds of columns, many of them irrelevant, give the model plenty of chance coincidences to latch onto.
- Testing on the training data, or on data that is too similar to it.
- Repeated tweaking against the same test set, until the model is tuned to that particular sample.
- Identifiers left in, such as customer numbers or dates, which let the model recognise individuals rather than patterns.
Overfitting's cousins: leakage and drift
Three different problems produce the same symptom, which is a model that is worse live than in testing. They have different fixes, so it helps to tell them apart.
| Problem | What went wrong | Typical fix |
|---|---|---|
| Overfitting | Model memorised noise in the training data | Simpler model, more data, fewer inputs, proper validation |
| Leakage | Training used information not available at prediction time | Rebuild inputs to use only what is known when the prediction is made |
| Drift | The world changed after training | Monitoring and regular retraining on recent data |
Leakage is, in our experience, even more common than classic overfitting in business projects. A model predicting cancellations using a 'reason for cancellation' field is a textbook case, and it happens more often than anyone likes to admit.
How to catch it before it costs you
- Hold data back. Keep a test set the model never sees during training or tuning, and look at it as few times as possible.
- Split by time. Train on, say, January 2023 to December 2025, and test on the first half of 2026. That mirrors how the model will actually be used.
- Compare training and test scores. A big gap between the two is the classic warning sign.
- Beat a simple baseline on the test set. If a complex model only just beats a simple rule, the complexity is probably memorising noise.
- Look at what drives predictions. If the most important input is a customer ID or an obscure internal code, something is wrong.
- Run a quiet pilot. Let the model make predictions live without acting on them for a few weeks, then compare with what happened.
Step six is the one we would never skip. It is inexpensive, it catches overfitting, leakage and integration bugs together, and it builds trust with the people who will use the predictions.
How it gets fixed
The remedies mostly involve giving the model less room to memorise:
- Choose a simpler model, or constrain a complex one so it cannot grow too detailed
- Remove inputs that should not matter, especially identifiers
- Add more data, particularly more varied data
- Use cross-validation, which tests on several different slices rather than one
- Stop training earlier for models that learn gradually
There is also the opposite problem, underfitting, where the model is too simple to capture real patterns. It shows up as poor scores on both training and test data. The sweet spot sits between the two, and finding it is a routine, unglamorous part of the job.
What to ask a supplier
When someone presents accuracy figures, ask how the test data was chosen, whether it came from a later time period, how many times the team looked at it while tuning, and what the training score was compared with the test score. A competent team answers in a minute. If the answers are vague, the figures probably are too.
At SpiderHunts we report training, validation and time-split test results side by side in every data science project, along with the baseline. If you are planning a deployment, our deployment guide covers the monitoring that catches drift once the model is live.
Frequently asked questions
How do I know if my model is overfitting?
Is overfitting always bad?
Does more data fix overfitting?
What is cross-validation in simple terms?
Model looked great in testing, disappointing in use?
Send us the details of how it was trained and tested. We will tell you whether overfitting, leakage or drift is the likeliest culprit and what it would take to fix.