Prompt Optimization BenchDo Auto-Optimized Prompts Beat a Hand-Written One
A benchmark pitting bootstrap few-shot and instruction search against a bare and a hand-written prompt on held-out ticket routing, showing which optimizer earns its cost and which one only overfits its dev pool.
Headline results
- Instruction search
- +6 points, matched the expert
- Few-shot bootstrap
- 1.000 dev, zero test lift
- What worked
- Optimising the instruction
- Model
- Real Claude Haiku
System architecture
Problem
Automatic prompt optimizers promise hand-tuned quality without the hand-tuning, and most write-ups report only the number the optimizer scored on the set it optimized against. That number is the one you cannot trust. The question worth answering is which method earns its build cost on a held-out test set, and whether the lift lives in the instruction or in the examples.
Approach
The task is routing support tickets to five internal teams under org-specific conventions a model cannot guess from general knowledge, so a slow dashboard goes to DataPlatform rather than Infra and a login failure after a deploy goes to Security. Gold labels follow from the conventions, so the accuracy delta measures the prompt rather than label noise. Four strategies compete on the same 33 held-out tickets: a bare zero-shot baseline, a human-written instruction spelling out the conventions, DSPy-style bootstrap few-shot, and APE-style instruction search. Both optimizers may only spend calls on a 15-ticket dev pool, and the report prints build calls next to test accuracy so a method's lift is judged against what it cost. The real run is Claude Haiku 4.5 on AWS Bedrock.
Impact
Instruction search hit 0.970 on test, +6 points over the 0.909 baseline, exactly tying the hand-written expert prompt for 79 dev calls. Few-shot bootstrapping scored a perfect 1.000 on dev and 0.909 on test, identical to the bare baseline, for 50 calls and zero lift. Same task, same model, same budget, and only one optimizer worked. The discipline that separates them is reporting dev accuracy next to test accuracy, without which the overfit few-shot result reads as a 1.000 win.
Decisions & tradeoffs
Report dev next to test, always
The few-shot optimizer's 1.000 dev score is the number a careless benchmark would publish. Printing dev and test side by side is what exposes the overfit, and it is the only reason the zero test lift is visible at all.
Pick a task with a rule the model cannot guess
Prompt optimization can only show attributable lift when the task has an org-specific convention that must be learned from examples or an instruction. Five teams with conventions that cut against the surface reading make the delta measure the prompt itself.
Charge every optimizer for its build calls
Both optimizers spend calls only on the dev pool, and the report shows that build cost next to test accuracy. Instruction search bought +6 points for 79 calls, few-shot bought nothing for 50, which is a judgment you cannot make from accuracy alone.
Build spec
- Model
- Claude Haiku 4.5 (Bedrock)
- Split
- 15 dev / 33 test, 48 tickets
- Instruction search
- 0.970 test (+0.061), 79 calls
- Auto few-shot
- 0.909 test (+0.000), 1.000 dev
- Stack
- Python, anthropic[bedrock], pytest
System notes
- Instruction search 0.970 test, tying the hand-written expert prompt for 79 dev calls
- Few-shot bootstrap: 1.000 dev, 0.909 test, zero lift over baseline
- Baseline already 0.909, so the honest headroom is small and stated
- Org-specific conventions make the lift attributable to the prompt, not label noise
Stack
Python · Claude · AWS Bedrock · Prompt Optimization · Evaluation · pytest