The most consequential line nobody discusses
A model produces a score; a threshold turns it into a decision. That threshold usually arrives as 0.5 because it is the default, and it is rarely the right answer.
It deserves as much attention as the model, because it is where the model meets the business. Two organisations using an identical model should often choose different thresholds, because their costs differ.
Put numbers on the two mistakes
Every threshold trades one error type against the other. Lower it and you catch more true cases and raise more false alarms; raise it and the reverse.
| Outcome | Example: flagging a fraudulent order | Cost to estimate |
|---|---|---|
| True positive | Fraud caught | Value of the loss avoided |
| False positive | Genuine customer blocked | Lost order, support time, goodwill |
| False negative | Fraud missed | Full loss plus handling |
| True negative | Genuine order passes | Nothing |
The estimates do not need to be precise. Order-of-magnitude figures agreed by the people who own the outcome are enough to move the threshold to a far better place than the default.
Compute and choose
- Take a held-out set with known outcomes and model scores.
- For each candidate threshold, count the four outcomes.
- Multiply by the agreed costs and total.
- Plot total cost against threshold - the minimum is your answer.
- Look at how flat the curve is near the minimum. A flat region means the exact value matters little, which is reassuring and lets you favour an easily explained round number.
Presenting that curve to stakeholders works better than presenting a number. People can see the trade-off and argue about the cost estimates, which is exactly the right argument to be having.
Several thresholds are often better than one
A single cut-off forces every case into act or ignore. Two thresholds create a middle band for human review, which usually fits reality better.
Size that band by review capacity. A review queue that generates more cases than the team can clear is not a control - it is a backlog, and the cases at the bottom are effectively unreviewed regardless of intent.
Who owns it, and when it changes
The threshold belongs to whoever owns the outcome - the fraud manager, the credit controller, the operations lead - not to whoever built the model. Their name should be attached to it in documentation.
It needs revisiting when the costs change: a margin change, a new customer acquisition cost, a change in how the team handles reviews. A threshold set once at launch and never revisited quietly stops being right.
The threshold is where the model becomes a business decision. Someone in the business should own it.