A better model, and an unhappy knowledge team
Your team moves a contract review feature to a newer model and adjusts the prompts. On your tests, results are better. A week later, a firm's knowledge lawyer writes: the risk ratings on indemnity clauses have changed, several clauses previously rated amber are now green, and their internal guidance for fee earners was written around the old ratings. Nobody told them. They want to know what changed and whether the old ratings were wrong.
Another firm has not noticed yet, but their innovation team ran a validation exercise on the old version last quarter and presented it to their risk committee.
Why firms treat output changes seriously
Law firms often validate a product before relying on it, and write guidance for fee earners about how to use it. That validation and guidance assume the product behaves consistently. In ordinary software, an improvement is simply good news. In a product whose outputs feed legal work, an unannounced change can undermine a firm's own controls.
- Model and prompt changes go to every firm at the same time.
- Release notes describe features, not changes in output behaviour.
- Firms cannot see old and new outputs side by side.
- There is no way for a firm to stay on the previous behaviour while it checks the new one.
- Your own tests show improvement overall, but not where outputs differ in ways a firm cares about.
What surprise changes cost
A firm that finds outputs changed without warning loses confidence in its own validation, and may suspend use of a feature until it re-checks. The knowledge team has to rewrite guidance at short notice. Your support team spends time explaining changes after the event. And the next time you want to improve a model, firms will push back, which slows the improvements your product depends on.
How we build changes firms can plan for
What we build lets you keep improving while each firm moves on its own terms.
- Versioned feature behaviour: each AI feature has a version made up of model, prompts and settings, and each firm is on a specific version.
- An evaluation comparison before release, run on your test set, showing where outputs differ between versions and by how much, grouped by clause type or question.
- Change notes written for knowledge lawyers: what changed, why, where outputs differ and examples, rather than a line in a release log.
- A preview mode where a firm can run the new version on its own sample documents and see old and new outputs side by side.
- Firm-controlled switch-over within a window you set, with a default date so firms do not stay on old versions for ever.
- A record per firm of which version produced each output, so past work can be explained after a switch.
| Stage | What the firm sees |
|---|---|
| New version ready | Change notes and comparison examples |
| Preview | Old and new outputs side by side on their documents |
| Decision | Switch now, or at the default date |
| After switch | Every output labelled with the version that produced it |
| Questions later | Record of version per output for past work |
Older versions are supported for a defined window, not indefinitely. That window is a commercial decision for you, and making it explicit helps firms plan.
The next model change
Your team prepares a new version of the contract review feature. The evaluation comparison shows most outputs unchanged and a group of indemnity ratings that differ, with examples. Change notes go to each firm's knowledge lead. One firm previews the new version on a set of its own contracts, agrees the new ratings are better, updates its guidance and switches early. Another waits for the default date. Nobody is surprised, and every output shows which version produced it.
Do your updates surprise firms?
- AI model or prompt changes go to every firm at once.
- Firms find out outputs changed from their own users.
- You cannot show a firm where old and new outputs differ.
- Firms cannot stay on the previous version while they check.
- Past outputs are not labelled with the version that produced them.