No. 232026Retrieval benchmark · Real Bedrock run

Long Context vs RAGFlat Cost or Linear Cost, Not a Winner

A needle plus multi-hop eval on a fictional corpus that sweeps context size and scores context-stuffing against RAG on accuracy, input tokens, and latency using real Claude Haiku on Bedrock.

RAG input tokens: Up to 11× fewer
RAG multi-hop: Falls to 0.50
Stuffing multi-hop: Holds
The axis: Flat vs linear cost

System architecture

Problem

Long-context models tempt teams to skip retrieval entirely and paste the whole corpus into the prompt. The instinct is rarely tested, because most needle-in-a-haystack demos use public documents the model may have memorized and stop at single-hop questions where retrieval never has to chain anything.

Approach

A fully fictional knowledge base of 15 needle passages padded with 90 on-topic distractors, so the model can only answer from what it was handed. Two methods run over the same pool at growing sizes: stuff puts all S passages in the prompt, and rag embeds the question with a MiniLM sentence-transformer retriever, pulls the top k, and prompts with only those. The corpus mixes single-hop questions answerable from one passage with multi-hop questions needing two passages chained, and gold passages are planted mid-list. Scoring is a normalized substring check, not an LLM judge, since every answer is a short fact and the model is told to say it does not know when the context lacks the answer, so a confident wrong answer scores zero. Real Claude Haiku 4.5 on AWS Bedrock, swept over pool sizes 10, 25, 50 and 80 at k=5.

Impact

RAG matched stuffing at up to 11x fewer input tokens. At 80 passages stuffing paid 2308 mean input tokens to RAG's 207 for the same single-hop accuracy, and both held 100% through 25 passages. But RAG's multi-hop accuracy fell to 0.50 at 50 passages and stayed there at 80 while single-hop stayed perfect, because top-k retrieval on the question text fetches the passage that looks like the question and not always the bridge passage. Stuffing's cost grows linearly with the corpus and RAG's is flat, so the real axis is flat-versus-linear cost, not a winner.

Decisions & tradeoffs

Build the corpus fictional

A public-document haystack cannot separate retrieval quality from memorization. Inventing every entity, company, and founder means a correct answer proves the passage reached the prompt, which is the only thing the benchmark is trying to measure.

Substring scoring instead of an LLM judge

Every gold answer is a short fact, a name, a year, or a term, and the model is instructed to answer in a few words or say it does not know. That makes a normalized substring check sufficient and removes a judge model that would otherwise confound a comparison between two retrieval methods.

State the scale caveat rather than overclaim

Stuffing never lost accuracy here, but the pools top out around 2300 tokens, far below real long-context scale where lost-in-the-middle degradation appears. The reproducible signals are the cost crossover and RAG's multi-hop drop as distractors grow, and the exact accuracy numbers are directional at 12 questions.

Build spec

Questions: 12 single-hop + multi-hop
Pool sweep: 10 / 25 / 50 / 80 at k=5
Token gap at 80: 2308 stuff vs 207 rag
RAG multi-hop at 50+: 0.50
Stack: Bedrock, sentence-transformers, pytest

System notes

RAG matched stuffing at 11x fewer input tokens: 207 vs 2308 at an 80-passage pool
RAG multi-hop accuracy fell to 0.50 at 50 passages while single-hop stayed 1.00
Fully fictional KB, 15 needles plus 90 distractors, so no pretraining leakage
Substring scoring, no LLM judge, with gold passages planted mid-list