Agent Eval HarnessTrajectory-Level Evaluation for Tool-Using Agents
An evaluation harness that measures whether a tool-using LLM agent actually completes the task, scoring task success, tool-call accuracy, step efficiency, and cost per task across a benchmark with reference traces.
Headline results
- Discrimination
- 100% vs 58% on one suite
- Metrics
- Success · trace F1 · steps · cost
- Agent
- Real Claude tool loop
- Inspector
- Streamlit trajectory viewer
System architecture
Problem
Everyone ships tool-using LLM agents, but almost no one can measure them. A final-answer-only metric hides how the agent got there, so a correct answer reached in 8 flailing steps and $0.40 looks identical to the same answer in 3 steps and $0.05. Teams need to grade the whole trajectory before trusting an agent in production.
Approach
The harness runs a tool-calling agent over a 12-task benchmark of multi-step tasks, each declaring a gold answer, an expected tool multiset, and an optimal step count. A single scorer computes task success (numeric-tolerant or text-substring), tool-call precision, recall, and F1 against the reference trace, step efficiency as optimal over actual steps, and USD cost per task. Three agents share that scorer: a real Anthropic Claude tool-calling loop, a key-free competent heuristic reference, and a deliberately flawed naive baseline. A deterministic in-memory tool world (calculator, catalog lookup, employee lookup, search) lets the harness, tests, and demo run fully offline. A CLI writes per-run JSON reports and a Streamlit viewer inspects per-task pass/fail and full trajectories.
Impact
On the same 12-task suite, the competent heuristic agent scores 1.00 success, 1.00 tool F1, and 1.00 step efficiency, while the deliberately flawed naive agent drops to 0.58 success, 0.59 mean tool F1, and 0.75 step efficiency. That the harness scores the flawed agent well below the competent one is the point: a good eval separates agents rather than rubber-stamping them.
Decisions & tradeoffs
Score the trajectory, not the answer
A final-answer-only metric hides how the agent got there. The scorer grades every tool call, step count versus optimal, and token cost so quality and efficiency are visible.
Prove the instrument discriminates
A deliberately flawed naive agent runs against the same scorer as a competent reference. Scoring it well below the competent agent demonstrates the eval separates agents rather than rubber-stamping them.
Key-free agents and a deterministic world
The heuristic and naive agents and an in-memory tool world let the harness, tests, and demo run fully offline. A real Anthropic Claude loop is available when an API key is present.
Build spec
- Benchmark
- 12 multi-step tasks with reference traces
- Metrics
- Success, tool F1, step efficiency, cost/task
- Agents
- Claude loop, heuristic, naive
- Interfaces
- ageval CLI + Streamlit trajectory viewer
- Stack
- Python, Claude, pytest, Streamlit, Docker
System notes
- Grades the full trajectory: tool-call F1, step efficiency, and cost, not just the final answer
- Discriminates a competent agent (100% success) from a flawed one (58%) on one suite
- 12-task benchmark with gold answers, expected tool multisets, and optimal step counts
- Real Claude tool-calling loop plus key-free agents that run fully offline
Stack
Python · Claude · Agents · Tool-use · pytest · Streamlit