Why this comes up
It seems theoretical until it is not. A customer disputes a decision from March. A bug is found and you need to know which predictions were affected. An auditor asks how a decision was made. A model degrades and you want to compare against the previous version.
In each case you need to reconstruct what the model was at a point in time, and in each case the answer is usually 'we think so' rather than 'yes'.
The four things that must be captured
| Component | What to record | Usual failure |
|---|---|---|
| Data | Exact training rows, or a snapshot reference | Source tables have moved on; cannot rebuild |
| Code | Commit hash of training and feature code | Usually fine, sometimes uncommitted local changes |
| Environment | Library versions, container image digest | 'Latest' pulled a new version silently |
| Configuration | Parameters, random seeds, thresholds | Held in someone's notebook |
Data is the one that defeats most teams. Operational tables update continuously, so re-running last March's query today returns different rows. Without a snapshot or a way to query as of a date, the model cannot be rebuilt.
Practical ways to pin the data
- Write the training set to immutable storage and record its location and checksum with the model
- Use a warehouse that supports time-travel queries, and record the timestamp
- Keep append-only history tables for key entities so state can be reconstructed as of a date
- At minimum, record the exact query and the run timestamp - weaker, but far better than nothing
For most businesses, writing the training extract to object storage alongside the model artefact is the simplest reliable option. Storage is cheap relative to the cost of not being able to answer.
What to store with each deployed model
Treat the deployed model as a package rather than a file. Alongside the weights, store the training data reference, the code commit, the environment digest, the configuration, the evaluation results on a named test set, the date and the person who approved it.
That package is what lets someone six months later answer questions without archaeology. It is also what a regulator or auditor will ask for, in more or less those terms.
Bit-exact or close enough
Perfect bit-level reproducibility is achievable but can be expensive - it requires pinning random seeds, controlling thread counts and sometimes accepting slower training.
For most business purposes, functional reproducibility is sufficient: rebuilding produces a model that behaves equivalently and makes the same decisions on a reference set. Decide which you need, and be aware that regulated contexts sometimes require the stricter version.
The code is almost always recoverable. The data as it stood that day usually is not.