The one-word edit that breaks production
Someone changes 'summarise briefly' to 'summarise concisely' because it reads better. Nobody tests it, because it is one word. A week later the support team notices the summaries have stopped mentioning refund amounts.
This is the most common way LLM features degrade, and it is entirely self-inflicted. Prompts behave like code, in that small changes have non-obvious effects, but they are often managed like copy, edited in a dashboard by whoever has access. The fix is not complicated. It is mostly habits borrowed from ordinary software engineering, applied to a new kind of artefact.
Versioning prompts properly
The first step is knowing exactly which prompt produced which output. Without that, every quality complaint turns into guesswork.
- Store prompts in the repository, as files or templates, reviewed through the same pull requests as code
- Give each prompt an ID and a version, and bump the version on any change, however small
- Pin the model alongside the prompt. A prompt version is only meaningful together with the model and parameters it was tested on
- Log the prompt ID and version with every call, next to the input and output
- Keep templates and variables separate, so a diff shows the instruction change, not a wall of customer data
Some teams prefer a prompt management tool so non-developers can edit wording. That is fine, provided the tool keeps history, supports a test run before publishing, and the app fetches a pinned version rather than whatever is latest. We compare the options in prompt management in production.
Building a regression test set
A regression set is a fixed collection of real inputs with a definition of what a good output looks like for each. It is the single most valuable asset in an LLM project, and it is usually smaller than people expect.
Fifty to two hundred cases is a sensible start for most business features. Pick them deliberately: the common cases, the awkward ones that caused complaints, the edge cases like empty inputs or foreign languages, and a handful of adversarial ones. Every production bug should add a case, which is how the set grows teeth over time.
| Output type | How to check it | Example |
|---|---|---|
| Structured fields | Exact or normalised match in code | Extracted invoice total equals expected |
| Classification | Label match, confusion matrix | Ticket routed to Billing |
| Must-include facts | Code checks for required items | Summary mentions refund amount |
| Tone and quality | Rubric scored by a model judge, spot-checked by people | Reply is polite and does not promise a date |
Prefer code checks wherever you can. A model grading another model is useful for fuzzy qualities, but it has its own error rate and needs calibrating against human judgement before you trust it.
Running tests on every prompt change
Once the set exists, wire it into the release process so nobody has to remember to run it.
- A prompt change opens a pull request like any other change
- CI runs the new version and the current production version against the full regression set
- The report shows the score for each, plus every case whose result changed
- A reviewer reads the changed cases, not just the headline number
- The change merges only if nothing important regressed
Reading the diffs matters. An overall score moving from 91% to 92% can hide five cases that got fixed and four that broke, and the four might be the ones that matter. Because model outputs vary between runs, run each case more than once for anything borderline and look at pass rates rather than a single result.
Releasing prompt changes safely
Passing tests is necessary but not sufficient. Real traffic always contains something your test set does not.
- Ship behind a flag and send a small share of traffic to the new version first
- Compare live signals between versions: user edits, thumbs-down, escalations, retries
- Keep the previous version one config change away
- Re-run the regression set when the provider updates the underlying model, even if you changed nothing
That last point catches teams out. A model update behind the same API name is effectively a change to every prompt you own, and the regression set is how you find out before customers do.
When this is overkill
Not every prompt needs a CI pipeline. An internal tool used by three people, where every output is read by a human before it goes anywhere, can live happily with a text file and common sense.
The investment pays off when outputs reach customers or systems without review, when several people edit prompts, or when the feature has enough volume that a quiet regression costs real money. For most customer-facing features that bar is met within the first month.
How SpiderHunts sets this up
On projects SpiderHunts builds, prompts live in the codebase from day one, every model call logs its prompt version, and the first sprint includes a regression set drawn from the client's real data. It costs a few days up front and saves the far more expensive conversation about why quality dropped and nobody knows when.
If you want the wider picture on measuring quality, how we decide whether an AI feature is good enough covers the thresholds, and our AI integration service includes the regression suite as standard.