Agent Memory EvalWhich Memory Strategy Survives a Long Conversation
A benchmark of four agent memory strategies (none, full history, summary, vector retrieval) on a multi-session recall test with real Claude Haiku on Bedrock, scored on recall accuracy against mean input tokens.
Headline results
- Retrieval recall
- 100% at ~1/3 the tokens
- Full history
- 100% at 462 tokens
- Summary
- Loses 40% of facts
- No memory
- 0%
System architecture
Problem
An agent that talks to a user across many sessions accumulates more history than fits in one context window. The field's answers are stuff everything in, summarize it, or retrieve the relevant part, and teams pick one on vibes. The question nobody measures is which strategy still answers a question about something said sessions ago, and what it costs in input tokens every single turn.
Approach
A multi-session history of 8 short sessions carries planted facts (a codename, an AWS region, a token rotation interval, a manager's name, a log retention period, an accent color), and 10 recall questions each target one planted fact. Four strategies run over the same history and the same questions: none (question only), full (entire history in the prompt), summary (history compressed once by the LLM into notes), and retrieval (history chunked per turn, embedded with all-MiniLM-L6-v2, top k=4 turns injected). Real Claude Haiku 4.5 on AWS Bedrock answers every question, and the benchmark records recall accuracy, mean input tokens per question, and the one-time setup cost of building the summary. Scoring is substring match against the planted fact and the agent is instructed to answer only from its notes and say it does not know otherwise, so a hallucination scores zero.
Impact
Retrieval matched full-history recall at a third of the cost, 1.000 accuracy on 159 mean input tokens versus 1.000 on 462 tokens, a 2.9x saving that does not grow with the conversation. Summarizing memory silently lost 40% of the facts at 0.600 accuracy, keeping the headline items but dropping exactly the specifics you recall later, the region us-west-2, the 30-minute rotation, the 400-day retention, and it cost an extra 550 setup tokens to be worse. No memory scored 0.000, confirming the task genuinely requires memory rather than priors. The honest ranking is retrieval over full over summary once the token bill is in view.
Decisions & tradeoffs
Plant checkable facts, score by substring
Every answer is a name, number, or short phrase planted in a known session, so correctness is decidable without an LLM judge. That removes the judge as a confound when the thing under test is memory.
Measure input tokens, not just accuracy
Full history ties retrieval on accuracy and looks fine until the per-turn token cost is counted. Reporting accuracy against mean input tokens is what surfaces retrieval as the winner and exposes summary as the trap.
A context-only fake agent for CI
The offline agent answers strictly from the context it was handed, modeling the real dynamic rather than faking the outcome. CI verifies the strategy mechanics with no keys and no network.
Build spec
- Strategies
- none, full, summary, retrieval
- Benchmark
- 8 sessions, 10 recall questions
- Model
- Claude Haiku 4.5 on AWS Bedrock
- Retriever
- all-MiniLM-L6-v2, top k=4
- Result
- retrieval 1.000 @ 159 tok, summary 0.600
System notes
- Retrieval matched full-history 1.000 recall on 159 vs 462 input tokens
- Summarization dropped 40% of planted facts, losing the specific numbers and names
- No-memory baseline scored 0.000, proving the task needs memory not priors
- Substring scoring plus an answer-only-from-notes prompt, so hallucinations score zero
Stack
Python · Claude · AWS Bedrock · Sentence-Transformers · RAG · Agents