The number that does not survive launch
The most common disappointment in forecasting work is a model that scored well in development and visibly worse in production. Usually nothing broke. The development score was simply measured in a way that could never be achieved live.
Backtesting done properly is the cheapest risk reduction available on a forecasting project. It costs compute time and no capital, and it tells you before launch roughly what to expect after.
Why a random split is wrong for time series
Standard machine learning practice shuffles rows and holds some out at random. For time series that is invalid, because it lets the model learn from Thursday in order to predict Wednesday. A real forecast has to be made without knowing what came next.
The damage is not subtle. A model with access to future periods can pick up the shape of a spike it should have had to predict, and report an accuracy that is simply unattainable. Teams then spend months wondering why live performance is worse.
Walk-forward validation, step by step
Walk-forward validation replays history in order. You choose a cut-off, train on everything before it, forecast the horizon you care about, score it, then move the cut-off forward and repeat.
- Pick your real forecast horizon. If you order six weeks ahead, test six weeks ahead, not one.
- Choose several origins spread across the history, covering different seasons and trading conditions.
- At each origin, train using only data available up to that date - including anything used to build features.
- Forecast the full horizon, score it, and record each origin separately rather than pooling into one average.
- Look at the spread across origins, not just the mean. A model that is excellent in three quarters and hopeless in the fourth is a different proposition from one that is steadily mediocre.
The leaks that hide in feature engineering
Even with correct splitting, information leaks in through features. These are easy to introduce and hard to spot once a pipeline is large.
- A rolling average computed over the whole dataset before splitting, so every training row has already seen the future.
- Category averages - mean sales per product - calculated across all time rather than up to the origin.
- Records updated in place: an order row whose status now reads 'returned' was not marked that way when the forecast would have been made.
- A price or cost field overwritten with the current value rather than the value that applied at the time.
The last two we find most often in business systems, because operational databases are built to hold current state, not history. If your ERP overwrites rather than versions, the honest answer may be that some features cannot be reconstructed for the past at all. Our note on data leakage goes further into how to spot it.
What the result should be compared against
A backtest score is meaningless alone. It needs a baseline computed over exactly the same origins and horizon, so the comparison is like for like.
Use the method the business actually uses today - the planner's spreadsheet, last year plus a percentage, or a seasonal naive forecast. If the model beats that by a margin too small to change any decision, that is a legitimate finding and much cheaper to learn before the build than after.
A backtest is not there to make the model look good. It is there to tell you what you are buying.
What to promise stakeholders
Live performance is usually a little worse than a careful backtest, even a clean one, because the world keeps changing in ways history did not contain. Presenting the backtest as a floor rather than a promise avoids an awkward conversation three months in.
We would rather show a client a modest, honestly measured improvement that holds than a flattering number that collapses. The second costs far more in trust than the first gains in enthusiasm.