Conformal PredictionCoverage You Can Guarantee
Wrap any trained classifier so its predictions come with a distribution-free, finite-sample coverage guarantee, and show why the softmax confidence people use instead cannot provide one.
Headline results
- 90% target
- 90.7% covered
- 95% target
- 94.9% covered
- Naive softmax
- 81.2% flat, misses both
- Cost
- Set grows 1.05 → 3.21
System architecture
Problem
When a decision-critical system needs to know how often it is right, teams reach for the softmax probability. That number is a story the model tells about itself. It carries no guarantee, and on an over-confident model it cannot deliver a coverage target no matter how you threshold it.
Approach
Split-conformal prediction, the LAC variant, wrapped around an already-trained classifier with no retraining and no distributional assumptions. One held-out calibration set the model never saw. The nonconformity score of the true label is one minus its predicted probability; take the finite-sample quantile of those scores, and the prediction set for a new input is every class scoring above the implied threshold. That set satisfies P(true label in set) at least 1 minus alpha, for any model and any distribution. The quantile correction is written from scratch.
Impact
On 1,500 test points behind a Random Forest that is 81.2% accurate, conformal hits every target: 81.9% coverage against an 80% goal, 90.7% against 90%, 94.9% against 95%. Naive softmax sits at 81.2% regardless, missing the 90% target by 8.8 points and the 95% target by 13.8. Conformal pays for the guarantee by growing the set on hard inputs, from 1.05 classes to 3.21, which is the honest cost and the reason the method is usable.
Decisions & tradeoffs
Report set size next to every coverage number
Coverage alone is trivially gameable by returning all classes. Publishing the mean set size at each target makes the trade explicit and is the difference between a guarantee and a technicality.
Run naive softmax as the control
The interesting claim is not that conformal works, it is that the thing people already use does not. Measuring both on the same test set turns that from an argument into a table.
Write the quantile correction by hand
The finite-sample correction is the one place where an off-by-one silently breaks the guarantee while every number still looks plausible. Implementing it directly makes it testable against hand-computed cases.
Build spec
- Method
- Split-conformal (LAC), distribution-free, finite-sample
- Model
- RandomForest at 81.2% accuracy, 6 noisy classes
- 80% target
- Conformal 81.9% ✓ · set 1.05 · naive 81.2%
- 90% target
- Conformal 90.7% ✓ · set 1.58 · naive 81.2% (−8.8)
- 95% target
- Conformal 94.9% ✓ · set 3.21 · naive 81.2% (−13.8)
System notes
- No retraining and no calibration assumptions: it wraps a model you already have
- The finite-sample quantile correction is implemented from scratch and unit-tested
- Set size is reported alongside coverage, because a guarantee bought with useless sets is not a result
- Naive softmax is run as a control and plateaus at the model's accuracy, which is the whole argument
Stack
NumPy · scikit-learn · Split-conformal · From-scratch quantile