LLM-as-Judge System800 Audited Scores, and an Honest Pending Calibration
An LLM-as-judge evaluation system: versioned rubrics, persisted chain-of-thought for every score, and a blind human-labeling protocol built to measure judge reliability rather than assume it. The judging run is complete; the human calibration is exported and pending.
Headline results
- Judged scores
- 800 (200 × 4 rubrics)
- Cost
- ≈$4.10
- Weakest rubric
- Factuality, mean 4.51
- Low scores
- 25 of 200 at ≤3
System architecture
Problem
An unmeasured judge is just one model's opinion. Teams adopt LLM-as-judge for scale, then never check whether the judge agrees with humans, where it's biased, or why it disagrees.
Approach
Scores 200 Claude Haiku outputs (stratified from AlpacaEval's 805 instructions) with a Sonnet judge on four versioned G-Eval rubrics: coherence, factuality, tone, safety, with anchored 1-to-5 scales and pinned evaluation steps. All 800 scores persist with full chain-of-thought. A blind annotation sheet, built so judge scores are hidden by construction, is exported to collect the human labels that Spearman agreement and a disagreement gallery will be computed from.
Impact
800 judged scores with auditable reasoning, at roughly $4.10. Factuality emerged as the weakest rubric (mean 4.51 vs 4.70 coherence, with 25 of 200 outputs scoring 3 or below) and open-ended tasks the hardest (4.28). The calibration itself is the honest gap: the protocol, the blind sheet and the scoring code are built, but the human labels have not been collected, so no judge-versus-human agreement number is claimed.
Decisions & tradeoffs
Pin the evaluation steps
G-Eval normally invents evaluation steps per run, which makes scores drift. Freezing rubric versions with pinned steps makes every score reproducible and every rubric change auditable.
Blind labels or no labels
Humans who can see the judge's score anchor to it, inflating agreement. The annotation sheet is built without judge output, so measured agreement is real.
Persist the chain-of-thought
A bare 3/5 is undebuggable. Storing the judge's reasoning for all 800 scores turns every disagreement into a diagnosable case instead of a mystery.
Build spec
- Dataset
- AlpacaEval · 200 stratified outputs
- Rubrics
- 4 G-Eval rubrics · anchored 1–5 scales
- Judge
- Claude Sonnet · temperature 0 · CoT persisted
- Calibration
- Protocol + blind sheet built · labels pending
- Scale
- 800 judged scores · ≈$4.10 · cached by rubric version
System notes
- 800 judged scores: 200 outputs by 4 rubrics, every score with stored chain-of-thought
- Versioned, frozen rubrics with pinned evaluation steps for reproducible scoring
- Blind calibration protocol: annotation sheets contain no judge output, preventing agreement contamination
- Calibration is stated as pending, not implied: no agreement number is reported until labels exist
What this does not show
- The judging run is done; the trust question is not. The blind calibration protocol and its 40-item annotation sheet are built and exported, but judge-versus-human agreement is pending labels, so the judge is not calibrated yet.
Stack
DeepEval · G-Eval · AlpacaEval · AWS Bedrock · SciPy · Python