Multi-Agent EvalWhen Does Multi-Agent Beat a Single Agent?
A benchmark that runs a planner to workers to critic multi-agent system against a single-agent baseline and quantifies the trade-off across quality, cost, latency, and call count.
Headline results
- Quality gain
- +5.5 points
- Cost
- 4.3×
- Latency
- 2.1×
- Verdict
- Not a free quality lever
System architecture
Problem
Multi-agent orchestration is one of the most over-hyped and under-measured areas in applied AI. Spinning up several agents feels sophisticated, but each one adds calls, tokens, and latency, and on many tasks the quality gain is marginal or zero. The senior question is not whether you used multiple agents but when the coordination overhead actually earns its keep.
Approach
Two systems run over the same task set: a single agent that answers the whole question in one call, and a multi-agent system where a planner splits the task, workers solve subtasks in parallel, and a critic synthesizes (2 plus number-of-subtasks calls). A key-point coverage scorer grades each answer against gold key points, and a harness (CLI maeval) aggregates quality, cost, latency, and call count into a verdict. Cost is real: total tokens times per-model pricing summed across every agent call, with latency modeled as planner plus max worker plus critic. Tasks are split into decomposable and simple so results can be read by category, not just on average. It runs fully offline with a calibrated key-free mock, and a live flag points it at Claude.
Impact
Across 24 tasks the multi-agent system averaged 0.639 quality versus 0.500 for single, but at 2.8x the cost and latency. The split is the real finding: on decomposable tasks multi-agent jumped 0.556 to 0.833 (+28 points), while on simple tasks it stayed 0.444 to 0.444 for zero gain at several times the cost. Multi-agent's quality edge is concentrated on genuinely decomposable tasks and vanishes on simple ones.
Decisions & tradeoffs
Read the split, not the average
Reporting only the aggregate 0.639 vs 0.500 would hide the real result. Splitting tasks into decomposable and simple surfaced that the entire quality edge lives in decomposable tasks and is zero on simple ones.
Real cost and latency accounting
Cost is total tokens times per-model pricing summed across every agent call, and latency models the multi-agent path as planner plus max worker plus critic with workers parallelized. This makes the 2.8x overhead a measured number rather than an assertion.
Key-free calibrated mock
The benchmark runs fully offline with a calibrated mock scored by lexical key-point overlap, so results are reproducible without an API key. The live flag runs the real pipeline on Claude to validate the pattern.
Build spec
- Tasks
- 24 (decomposable + simple)
- Multi-agent quality
- 0.639 vs 0.500 single
- Decomposable gain
- +28 pts (0.556 to 0.833)
- Cost/latency overhead
- 2.8x
- Stack
- Python, Claude tool loop, pytest
System notes
- Planner to workers to critic vs single-agent baseline on the same tasks
- Decomposable tasks: +28 quality points (0.556 to 0.833)
- Simple tasks: zero gain at 2.8x cost and latency
- Runs fully offline via a calibrated key-free mock; live flag for Claude
What this does not show
- The mock's clean decomposable-only story did not reproduce on real Claude Haiku; it inverted, within noise.
Stack
Python · Claude · Multi-agent · Evaluation · pytest · Docker