Expect some drop, investigate a large one
A modest decline from test to production is normal. Development data is cleaner, the period is fixed, and any tuning has fitted the evaluation somewhat.
A large drop is a defect, and it has a findable cause. Five causes cover almost everything we see.
One: leakage in training
The most common cause. The model had access during training to something unavailable at prediction time, so its test score was never achievable.
The tell is a test score that seemed surprisingly good. If a feature made the model dramatically better, examine when that value is actually populated - fields written during or after the event you are predicting are the usual culprits.
Two: a different pipeline in production
Training runs on a carefully prepared extract; production runs on live data through different code. Small differences change results in ways that are hard to see.
- A category encoded differently, so unseen values fall into a default bucket
- Units differing - pence against pounds, or a date parsed day-first in one place and month-first in the other
- A join producing duplicates in one path and not the other
- Missing values filled differently, or not at all
- Rounding or type conversion applied in one pipeline only
The direct check is to score the same records through both paths and compare outputs row by row. Any difference beyond floating-point noise is a bug, and this test finds it in an afternoon.
Three: the population has changed
The model was trained on last year's customers, orders or equipment, and this year's are different. A new channel, a marketing push into a new segment, a price change, or simply growth into a different market.
Compare the distribution of key inputs between training and live. Where they have moved materially, the model is being asked about a population it never saw. Retraining on recent data is usually the answer, and monitoring input distributions is how you catch it earlier next time.
Four and five: feedback, and measuring differently
Feedback is subtle: the model's actions change the data it later learns from. Decline the risky applications and future data contains only accepted ones, so the model never learns about the group it rejected. A random holdout that bypasses the model is the standard defence.
The last cause is the most embarrassing and worth ruling out early. Sometimes the model is fine and the production metric is computed differently - a different denominator, a different time window, a different definition of success. Reconcile the two calculations on the same records before assuming a modelling problem.
Before rebuilding the model, check that both numbers were measuring the same thing.