Human Review Queues for Low-Confidence Predictions

Full automation is usually the wrong first goal

A logistics firm wants a model to classify incoming delivery exceptions into 14 categories. The model is right about 91 per cent of the time. Nobody is comfortable letting it act on its own, so the project stalls while people argue about whether 91 is good enough.

It is the wrong argument. Split the predictions by confidence and the picture changes: for the most confident two thirds, accuracy is near perfect. The uncertain third is where almost all the errors live. Automate the first group, send the second to a person, and you have a working system this month rather than a debate for the next quarter.

How a review queue works

1. The model makes a prediction with a confidence score.
2. Predictions above the threshold are acted on automatically and logged.
3. Predictions below it go into a queue with the model's suggestion and reasons.
4. A reviewer accepts, corrects or escalates each item.
5. The decision is recorded and later used to evaluate and retrain the model.
6. A random sample of automated predictions is also reviewed, to check the threshold still holds.

Step six is the one that gets forgotten. Without it you only ever see the cases the model was unsure about, and you will not notice if it becomes confidently wrong.

Setting the confidence threshold

The threshold should come from data, not from a round number that feels safe. Take a labelled set of recent predictions, group them by confidence, and measure accuracy in each band.

The numbers above are an example, not a benchmark. The pattern is typical though: accuracy climbs steeply with confidence, if the scores are well calibrated. Check calibration before trusting them. Many models are overconfident out of the box and need calibrating so that a 0.9 really does mean right about nine times in ten.

Then bring in cost. If a wrong automated decision costs a customer refund of a few pounds, a lower threshold is fine. If it means a wrongly released payment, set it high and accept a bigger queue.

Sizing the queue to your people

A threshold that sends 400 items a day to a team that can review 150 is not a design. It is a backlog. Work it out before launch.

• Daily volume multiplied by the share below threshold gives queue size
• Time per review, measured on real items, gives hours needed
• Allow for peaks, such as Monday mornings or seasonal spikes
• Decide what happens when the queue overflows: a default action, a temporary threshold change, or overtime
• Set a target time for items to be reviewed, and alert when it slips

As the model improves, the queue should shrink. Track its size weekly; it is one of the clearest measures of whether the investment is paying off.

Designing the review screen

Review speed depends far more on the interface than on the model. A reviewer switching between three systems to check a case will manage a fraction of what they could with everything on one screen.

• Show the original item, the model's suggestion and its main reasons together
• Make accepting the suggestion one click or one key
• Offer the most likely alternatives as quick choices when correcting
• Capture a short reason code for corrections, not free text
• Hide the confidence number from reviewers if it makes them rubber-stamp

If reviewers accept 99 per cent of suggestions in under two seconds, they are not reviewing. Either lower the queue or change the screen.

Our post on building the review interface for an AI app goes deeper into layout, and showing why a model made a decision covers what reasons to display.

Learning from reviewer decisions

Every corrected prediction is a labelled example of exactly the kind of case the model struggles with. That is valuable training data, and it is sitting in the queue for free.

Two cautions. First, reviewers can be wrong or inconsistent, so check agreement between reviewers on a sample. Second, the queue only contains low-confidence items, so do not train or evaluate on it alone or you skew the model towards hard cases. Combine it with the random sample of automated decisions.

When a review queue is the wrong answer

If almost everything lands below the threshold, the model is not ready and a queue just adds a step to manual work. If reviews need the same effort as doing the task from scratch, automation is saving nothing. And for decisions where the law requires meaningful human involvement, a queue that only sees some cases may not be enough.

SpiderHunts designs review queues into most automation and machine learning projects that touch money or customers. It is how clients get value in weeks while trust in the model is still being earned. For agent-based systems the same idea applies at the level of actions; see human-in-the-loop design.

Frequently asked questions