The assumption that fails
Ordinary k-fold cross-validation shuffles the data and rotates which portion is held out. It assumes each row is independent of the others, which is reasonable for, say, a set of images and false for anything with a time order.
With time-ordered data, shuffling puts January and March in training while February is held out. The model sees what happened on both sides of the gap, which is information no real prediction would have.
Forward chaining instead
The correct structure trains on a period and validates on the period immediately after, repeatedly, moving forward.
| Fold | Train on | Validate on |
|---|---|---|
| 1 | Months 1-12 | Months 13-15 |
| 2 | Months 1-15 | Months 16-18 |
| 3 | Months 1-18 | Months 19-21 |
| 4 | Months 1-21 | Months 22-24 |
Each fold reflects a realistic situation: everything known up to a point, predicting what comes next. The scores can be averaged, but the spread across folds is at least as informative as the mean.
Sliding or expanding window
The table above uses an expanding window - training data grows each fold. The alternative is a sliding window of fixed length, dropping the oldest data as it moves.
- Expanding suits stable relationships where more history helps.
- Sliding suits businesses that change, where old data actively misleads.
- Testing both answers a genuine question: does old data still help? If sliding wins, that is evidence your business has changed.
That comparison is worth running regardless of which you adopt, because it tells you something about the business rather than only about the model.
Gaps, and why you may need one
If predictions are made some distance ahead of when they are used, the validation split should reflect that. Forecasting six weeks out means the last six weeks before the validation period would not have been available.
Leaving that gap makes the test harder and more honest. Skipping it produces a score achievable only if data arrives instantly, which it does not.
Keep one period genuinely untouched
Cross-validation folds get used repeatedly while tuning, and tuning against them gradually leaks their information into the model's design. By the twentieth configuration, the score is optimistic.
Hold back a final period that is not looked at until you believe the work is finished, and check once. If that result disappoints, the honest response is to accept it rather than to keep adjusting until the number improves.
A test set you have optimised against is not a test set any more.