Calibration EvalDoes the Model Know When It Is Wrong
An ECE and reliability-diagram audit of Claude Haiku's self-reported confidence against actual correctness on a graded-difficulty quiz whose answers are computed rather than looked up.
Headline results
- Aggregate ECE
- 0.12
- Expert tier
- 80% accurate at 96% confident
- 98% confidence bin
- 93% accurate
- The point
- The aggregate hides it
System architecture
Problem
Systems route, escalate, and auto-approve based on a model's self-reported confidence. That only works if the confidence is calibrated: when the model says it is 90% sure, it should be right 90% of the time. Almost nobody checks, and the aggregate number that gets quoted turns out to hide exactly where the risk is.
Approach
Every question is a computed arithmetic or logic problem rather than a lookup, because calibration needs ground truth you can trust absolutely and one wrong gold answer poisons the reliability curve. Distractors are plausible near-misses and difficulty is graded across four tiers, from two-digit sums the model nails to multi-step and modular problems it genuinely fails, so correctness actually varies. Real Claude Haiku 4.5 on AWS Bedrock answers 60 questions, 15 per tier, reporting an answer and a confidence from 0 to 100. ECE, reliability bins, overconfidence, and Brier score are implemented from scratch and unit-tested against synthetic arrays with known values.
Impact
The aggregate looks healthy and hides the whole story: overall accuracy 0.933, mean confidence 0.905, overconfidence -0.028, ECE 0.121. The per-tier split is where the risk lives. On the expert tier the model scored 0.800 accuracy at 0.961 mean confidence, a +0.161 overconfidence gap on precisely the questions it fails most, which is the dangerous direction. 55 of 60 answers landed in the top confidence bin at 0.978 average confidence but only 0.927 accuracy, so 98% sure really means about 93% right. At the other end it is erratically underconfident, returning 5 to 15% confidence on easy items it got right and dragging that tier to -0.252. The middle is genuinely calibrated, medium -0.017 and hard -0.004, which is exactly what makes the average look fine.
Decisions & tradeoffs
Compute the answers, never look them up
Calibration is only as trustworthy as the gold labels, and a single wrong answer distorts the whole reliability curve. Generating arithmetic and logic problems makes ground truth exact and lets difficulty be dialed to force real variance in correctness.
Report per-tier, not just aggregate
The overall -0.028 overconfidence is two opposite errors cancelling out. Splitting by difficulty is what exposes the +0.161 overconfidence on the expert tier, which is the number a downstream system would actually be hurt by.
Parse the last stated confidence
The model sometimes writes a confidence, recalculates, and revises it. Taking the final value scores what the model actually concluded rather than penalizing it for reasoning out loud.
Build spec
- Quiz
- 60 computed MCQs, 15 per tier
- Tiers
- easy, medium, hard, expert
- Model
- Claude Haiku 4.5 on AWS Bedrock
- Overall
- acc 0.933, ECE 0.121, Brier 0.130
- Expert tier
- acc 0.800 vs conf 0.961 (+0.161)
System notes
- Overconfident where it fails most: expert tier 0.800 accuracy at 0.961 confidence
- The 0.9 to 1.0 confidence bin holds 55 of 60 answers but is only 0.927 accurate
- Underconfident and noisy on trivial items, dragging the easy tier to -0.252
- ECE, reliability bins, overconfidence, and Brier implemented from scratch and unit-tested
Stack
Python · Claude · AWS Bedrock · Calibration · ECE · Evaluation