Confidence intervals that are not
Many models can output an interval, and most of those intervals rest on assumptions - a particular error distribution, constant variance - that business data rarely satisfies. The result is an interval that sounds rigorous and does not cover what it claims.
That matters when someone plans against it. An interval claiming 90% coverage that delivers 60% is worse than no interval, because it invites false confidence.
How conformal prediction works
The idea is simple enough to state in a paragraph. Train the model as usual, then set aside a calibration set the model has not seen. Measure how wrong the model was on each calibration case. To predict on a new case, use the distribution of those past errors to set the interval width.
- Train on the training set.
- Predict on a separate calibration set and record the errors.
- Choose a coverage level, say 90%.
- Find the error value below which 90% of calibration errors fall.
- For a new prediction, the interval is the point prediction plus or minus that value.
Under mild assumptions, that interval covers the true value about 90% of the time - and the guarantee does not depend on the model being correct, only on new data resembling the calibration data.
Where it is genuinely useful
- Deciding what to review. Wide intervals mark cases the model finds hard - a better routing rule than confidence scores from the model itself.
- Planning against a range. An interval with real coverage can be used for stock or capacity decisions.
- Telling people the truth. Showing a range rather than a point sets expectations correctly and survives an inevitable miss.
- Detecting drift. If observed coverage falls below the target, something has changed.
That last one is a useful by-product. Coverage is measurable as outcomes arrive, so the guarantee doubles as a monitoring signal.
Adaptive intervals
The basic version gives every prediction the same width, which is honest but blunt. Adaptive variants scale the interval by how difficult each case looks, giving narrow intervals on easy cases and wide ones on hard ones.
That is usually what you want operationally, because it separates cases worth a human look from cases that can proceed automatically - and it does so on a principled basis rather than an arbitrary score threshold.
What it does not do
The guarantee assumes new data resembles the calibration data. Under genuine distribution shift it degrades like everything else, so it is not protection against the world changing.
It also does not make the model better. A poor model produces wide intervals - correctly so. That is arguably its most useful property: it makes model weakness visible rather than hiding it behind a confident point estimate.
A wide interval is not a failure. It is the model telling you it does not know, which is worth a great deal.