Where to Put the Person in an Automated Process

Two different reasons for a checkpoint

• Uncertainty: the system is not confident, so a person decides
• Consequence: the system is confident, and the decision matters too much to leave unreviewed

The first is tunable. The second is a policy decision and should not move with the accuracy figures.

Setting the confidence threshold

1. Run in shadow and record confidence against correctness on real cases
2. Find the level above which errors are acceptably rare
3. Set the threshold there, then move it slightly conservative
4. Review monthly against actual outcomes

A threshold set by intuition is a guess with a decimal point. Set it from a shadow run and it means something.

Always sample the confident items

A proportion of high-confidence output should be reviewed forever, even when it is always right. It is how you detect drift before users do.

Two to five per cent is usually enough, and it is the cheapest quality insurance in the system.

Decisions that stay human regardless

• Anything that spends money or commits to a price
• Anything a customer receives as a decision about them
• Anything difficult to reverse
• Anything a regulator expects a named individual to have done

Make the checkpoint fast or it fails

A checkpoint that takes two minutes per item becomes a rubber stamp within a fortnight. Ten seconds per item keeps it real.

That is a design constraint on the interface, and it should be measured after launch rather than assumed.

Frequently asked questions