The switch comes first
Before monitoring, before evaluation, before anything: a named person in your business can turn the AI step off, from a screen they can reach, in under a minute.
When output is going wrong at volume, the ability to stop it matters more than the ability to diagnose it. Diagnosis can wait an hour; the wrong output cannot.
What the switch should do
- Stop new AI processing immediately
- Fall back to the manual path, not to nothing
- Leave already-queued work in a reviewable state
- Notify whoever needs to know, automatically
- Be reversible just as quickly
Degrade before you stop
A middle setting is often better than off: route everything to human review rather than halting. The work still moves, with a person in every loop.
That is the setting most incidents actually need, and it is trivial to build if the confidence threshold is already configurable.
Version everything that changes behaviour
- Prompts versioned, with the previous one one click away
- Model version recorded on every output
- Threshold changes logged, with who and when
- The index state recorded, so an answer can be traced to a document version
Rollback is only possible if you know what changed. Most AI incidents are a change nobody recorded.
Rehearse it
Once, before launch, have the business owner actually use the switch and watch the fallback work. A control nobody has tested is a control nobody will use at the moment it matters.
It takes twenty minutes and it changes how people feel about the whole system.