Drift and structural change are different
Drift is gradual: customers change, seasons shift, behaviour moves. Monitoring is designed for it and handles it reasonably.
A structural change is abrupt and different in kind. A new pricing model, a migrated ERP, an acquired business, a changed product hierarchy - after these, history describes a business that no longer exists.
Monitoring may detect the effect eventually, but by then decisions have been made. These changes are known in advance, which means they can be planned for.
Changes that should trigger a review
- Pricing or discount structure changes
- A system migration, particularly where field definitions change
- An acquisition or merger bringing different customers or products
- A product hierarchy or category restructure
- A significant channel change - opening online, closing branches
- A regulatory change altering what can be collected or how decisions are made
- A major process change in how work is recorded
System migrations cause the most damage per unit of attention received. A field that meant one thing in the old system and something subtly different in the new one produces a model that is confidently wrong with no visible error.
What to do, in order
- Identify which models depend on what changed - which is much easier if you maintain a register of models and their inputs.
- Decide whether history remains valid. Sometimes it can be mapped forward; sometimes it genuinely cannot.
- Where it can be mapped, apply the mapping consistently to history and document it.
- Where it cannot, consider training on the shorter post-change period, accepting reduced data.
- Increase monitoring frequency around the change, and keep the previous model available for comparison.
Point one depends on having a model register. Organisations with several models frequently cannot answer which ones use a particular field, which makes every change riskier than it needs to be.
The awkward period afterwards
| Option | Trade-off |
|---|---|
| Keep using the old model | Fast, but increasingly wrong |
| Retrain on mapped history | Good if mapping is sound; risky if not |
| Retrain on post-change data only | Correct but little data at first |
| Fall back to a rule temporarily | Crude, predictable, buys time |
| Blend old and new | Complex but smooths the transition |
There is no free option. The right choice depends on how fundamental the change was and how much post-change data you can accumulate before decisions have to be made.
Get into the change conversation earlier
The recurring problem is that model owners hear about business changes after they happen. Systems migrate, pricing changes, and nobody thinks to mention it to whoever maintains the forecast.
The fix is organisational: get models represented in whatever change process exists, and maintain a register that makes the dependency visible. A single line in a project checklist asking which models are affected prevents most of this.
After a system migration, your history describes a business that no longer exists.