Treating Prompts as Configuration

Prompts change more often than code

A prompt is adjusted when output quality shifts, when the business changes its terminology, or when a provider updates a model. That happens far more often than the surrounding code changes.

A prompt embedded in a function requires a deployment to change. In configuration, it requires an evaluation run and a configuration update, which is the right level of ceremony.

What good management looks like

1. Versioned, with a history of what changed and why
2. Tested against the evaluation set before going live
3. Rollback in one action to the previous version
4. Recorded on every output, so results can be explained
5. Reviewable by someone who understands the business, not only a developer

Test every change

• Run the full evaluation set before deploying
• Compare the score against the current version
• Look at what changed, not just the total
• Block the change if the score drops beyond normal variation

Prompt changes have side effects. Improving one behaviour frequently degrades another, and only the evaluation set reveals it.

Record the version on every output

When an output is questioned months later, the answer requires knowing which prompt and which model produced it. Recording both costs nothing and is the difference between explaining and guessing.

It is also what lets you correlate a quality change with the change that caused it.

Keep them readable

Frequently asked questions