Hallucination Detection PipelineA Finished Pipeline, an Unfinished Benchmark
A complete, resumable TruthfulQA hallucination pipeline on Bedrock: dual-mode scoring, per-response caching, throttle-resilient. Validated end to end on a 20-question pilot; the full 817-question run and per-topic breakdown are still pending.
Headline results
- Pilot
- 20 questions
- Haiku
- 10% MC · 20% free-form
- Sonnet
- 15% MC · 15% free-form
- Generated
- 817 for Haiku
System architecture
Problem
Hallucination isn't random: models fail far more on misconception-prone domains like health and law. Aggregate scores hide exactly the failures that matter, so deployed systems need per-topic rates, not one number.
Approach
A complete, resumable pipeline (load, generate, score, analyze, report) built to run TruthfulQA's full 817-question, 38-category benchmark through Claude Haiku and Sonnet in two modes: multiple-choice with judge-free exact-match scoring, and free-form generation judged by a strictly stronger model. Per-response caching keyed by model and question means an interrupted run resumes with zero re-spend. So far it has been exercised end to end on a 20-question pilot, not the full benchmark.
Impact
The 20-question pilot already surfaced a mode gap invisible to standard leaderboards: Haiku hallucinated 10% on multiple-choice but 20% free-form, so a model that picks true answers can still generate false ones. That is a pilot, not a benchmark. The full 817-question run has not been completed and no per-topic rates exist yet, which is stated here rather than implied away: this is a finished pipeline waiting on a run that costs about $19.
Decisions & tradeoffs
Category buckets over aggregate scores
One hallucination number tells you nothing actionable. Bucketing 38 categories into seven domains turns the result into a deployment decision: which topics this model can't be trusted on.
Two evaluation modes, deliberately
Multiple-choice and free-form measure different things, and the gap between them is itself a finding. A model that picks true answers but generates false ones fails differently in production.
Judge stronger than judged
Using a stronger model to judge keeps judge errors from masquerading as candidate hallucinations. Judge reasoning is persisted for every verdict.
Build spec
- Status
- Pipeline complete · 20-question pilot only
- Benchmark
- TruthfulQA · 817 questions · 38 categories loaded
- Candidates
- Claude Haiku + Sonnet (Bedrock)
- Judging
- MC exact-match + LLM judge, free-form
- Pending
- Full run ≈$19 · per-topic breakdown
System notes
- Pilot measured: Haiku 10% multiple-choice vs 20% free-form, the same model failing differently by mode
- Judge strictly stronger than candidates to reduce judge-capability bias
- Per-response caching keyed by model and question: interrupted runs resume free
- Scope stated honestly: 20 of 817 questions scored, so no per-topic rate is claimed
What this does not show
- Only the 20-question pilot is scored, and a 20-question pilot is not statistically meaningful. The full 817-question run and the per-topic breakdown are pending, so no per-topic failure rate should be quoted.
Stack
TruthfulQA · DeepEval · AWS Bedrock · Claude · HuggingFace · Python