The problem appears around month three
Early on, prompts live wherever they were written. Then a model updates, output quality shifts, somebody tweaks a prompt to fix it, and something else breaks that nobody noticed for a fortnight.
By then there are prompts in six files, no record of what changed, and no way to tell whether last week's edit helped or harmed.
Four practices that prevent it
- One place. All prompts in a single location, not scattered through application code.
- Versioned. In source control, with a version identifier recorded on every output.
- Tested. An evaluation set that runs before any prompt change ships.
- Reviewed. Prompt changes go through the same review as code changes, because they have the same blast radius.
Log the version with the output
When someone disputes an answer the system gave three weeks ago, you need to know which prompt and which model produced it. Without that, investigation is guesswork.
Store prompt version, model identifier, retrieved context references and the output. Storage is cheap and this record is what makes debugging possible.
Separate the parts that change
A prompt usually has three components: stable instructions, retrieved or supplied context, and the user's input. Keep them structurally separate.
This also matters for cost: putting the stable part first makes prompt caching effective, which can be a substantial saving at volume.
Change one thing at a time
Prompt changes interact. Adjusting instructions and swapping models in the same release means you cannot attribute the difference in behaviour.
Ship one variable at a time, run the evaluation set, record the result. Slower, and it is the difference between improving deliberately and changing things hopefully.
Who is allowed to edit them
There is a temptation to let non-technical staff edit prompts directly, since they are just text. That works only with the same safeguards code has: versioning, testing and review.
A reasonable arrangement is that anyone can propose a change and the evaluation set decides whether it ships.