LLM Judge BiasAuditing the Judge That Scores Everything Else
An audit of the LLM-as-judge itself for position, verbosity, and assertiveness bias, judging controlled answer pairs in both slot orders with real Claude Haiku on Bedrock.
Headline results
- Position bias
- None, order-invariant
- Verbosity bias
- 75%
- Assertiveness bias
- 100%
- Implication
- Judge scores are gameable
System architecture
Problem
LLM-as-judge is the backbone of modern evaluation, from leaderboards to RLAIF reward signals. It also has biases that inflate scores for reasons that have nothing to do with answer quality. If the judge rewards length or tone, every number built on top of it is measuring style as well as substance, and nobody downstream can tell the difference.
Approach
Three controlled pair sets are engineered so a fair judge has an obvious, order-independent answer, and any systematic deviation is the bias. Position pairs put a genuinely correct answer against a genuinely wrong one, verbosity pairs put a concise answer against an equally correct one padded with true-but-filler detail, and assertiveness pairs state identical correct facts hedged versus confidently. Real Claude Haiku 4.5 on AWS Bedrock judges all 32 pairs twice, once with each answer in each slot, producing 64 judgments. Both-order judging separates first_slot_rate (position preference, where 0.50 means none) from pick_worse_rate (how often the judge took the wrong, longer, or confident answer) and flip_rate (how often swapping the order reversed the verdict).
Impact
The judge has no position bias at all. On pairs with a clear correct answer it picked the wrong one 0% of the time, took the first slot exactly 0.500 of the time, and never flipped when the order was swapped. The bias is entirely in style. Verbosity bias ran to 0.750, a 75% preference for the padded answer where fair is 0.500, and assertiveness bias was total at 1.000, the confident phrasing winning in every single pair in both orders. The consequence is uncomfortable and concrete: a model under evaluation can lift its judge score by being longer and more assertive with zero improvement in substance, which is why order-swapping, length controls, and calibration against human labels are prerequisites for trusting a judge.
Decisions & tradeoffs
Judge every pair in both orders
One pass conflates position preference with the correct verdict. Running each pair twice with the slots swapped makes first_slot_rate and flip_rate measurable independently of whether the judge got the answer right.
Engineer pairs where fair behavior is known
Each pair has a defined fair value, 0.00 pick-worse for position and 0.50 for verbosity and assertiveness. Without a known fair baseline a rate is just a number, not a bias.
A fake judge with an injected bias
The offline judge is deliberately biased in a known direction and the tests assert the metrics recover it. This validates the bias arithmetic itself rather than trusting that the math is right.
Build spec
- Biases
- Position, verbosity, assertiveness
- Pairs
- 12 / 10 / 10, each judged both orders
- Judge
- Claude Haiku 4.5 on AWS Bedrock
- Result
- position 0.000, verbosity 0.750, assertiveness 1.000
- Cost
- 7,694 input / 1,636 output tokens
System notes
- No position bias: 0.000 pick-worse, 0.500 first-slot, 0.000 flip rate across 12 pairs
- 75% verbosity bias toward the padded but equally correct answer
- 100% assertiveness bias for confident over hedged phrasing of identical facts
- Every pair judged in both slot orders, isolating bias from the correct verdict
Stack
Python · Claude · AWS Bedrock · LLM-as-Judge · Evaluation · Bias