Codegen EvalVerifying What The AI Wrote
An eval harness for AI-generated code: which prompting and verification strategies catch which classes of failure, measured on both a ground-truth corpus and a live Claude run.
Headline results
- unit layer
- 100% across all 5 classes
- mutation · llm_review
- 0% across all 5
- Real run
- 40 solutions · $0.0160
- False flags
- 0 of 26 clean
System architecture
Problem
Interviews changed. Meta runs AI-assisted coding rounds, Google added a round on comprehending Gemini-generated code, Canva grades the judgment layered on top of AI tools. The skill under test is no longer writing a binary search, it is knowing when the AI's binary search is wrong, insecure, or quadratic. That skill has no benchmark.
Approach
Cross four prompting strategies with five verification layers and measure which combination catches which class of failure. The strategies are bare, spec, test-first, and self-review. The layers are unit tests in a sandbox, property checks over seeded random inputs, mutation testing, lint plus a builtin insecure-pattern scanner, and LLM review. A seeded mock corpus with known planted defects establishes ground truth for the catch-rate matrix; a real run puts Claude Haiku 4.5 through the same battery.
Impact
Two results worth having. On the seeded corpus, unit tests catch 100% of every failure class while mutation testing and LLM review catch nothing at all, with zero false flags anywhere. On 40 real Haiku solutions, test-first produced 0 flagged defects out of 10 and bare produced 2, but the sharper finding is that self-review rewrote its code on the second pass and still shipped the same specification violation. The mock's planted eval() danger never occurred on the real model; the real defect was an edge case that only property testing saw.
Decisions & tradeoffs
Report the layers that caught nothing
Mutation testing and LLM review score 0% across every failure class in this corpus. Dropping them would make the matrix look better and teach less. A verification battery is only useful if you know which parts of it are not earning their runtime.
Let the real model overturn the mock
The mock corpus seeds an eval() vulnerability that the lint layer reliably catches. Real Haiku never used eval, exec, or a shell in any of 40 solutions, so the layer built for that threat found nothing and the real bug was elsewhere. The seeded story and the measured story are both kept.
Execute generated code in a sandbox, always
Every unit and property run happens in a sandboxed subprocess with timeouts. Benchmarking untrusted generated code by importing it into the harness process is how a benchmark becomes an incident.
Build spec
- Strategies
- bare · spec · test-first · self-review
- Layers
- unit · props · mutation · lint · llm_review
- Real run
- 40 solutions · 50 Bedrock calls · 19,963 tokens · $0.0160
- Real defects
- bare 2/10 · spec 1/10 · test-first 0/10 · self-review 1/10
- False flags
- 0 of 26 clean solutions, every layer
System notes
- Property testing over seeded random inputs caught a slugify bug that the shown functional tests could not see
- Self-review costs roughly 2× and did not fix its own spec violation, which is the case against trusting a model to check itself
- Zero false flags across all five layers on 26 clean solutions, so the catch rates are not bought with noise
- The catch-rate matrix is byte-identical whether or not bandit and ruff are installed
What this does not show
- The mock's seeded eval() vulnerability never occurred on the real model, so the lint layer built for it found nothing and the real defect was elsewhere.
Stack
Claude on Bedrock · hypothesis · mutmut · bandit / ruff · Sandboxed exec · pytest