Production RAG LabEvery Layer of RAG, Benchmarked
A config-driven harness that benchmarks every layer of production RAG on one corpus, one eval set, one metric suite. 11 of 17 phases complete, generation on real Claude Haiku, with the honest results the tutorials skip.
Headline results
- Phases
- 17 of 17 built
- Rerank on a weak stage
- recall@1 0.55 → 0.95
- Rerank on a strong stage
- no gain, 2.7× latency
- Tests · runs
- 178 · 110
System architecture
Problem
RAG is easy to prototype and hard to productionize. Every layer, chunking, embedding, indexing, retrieval, query understanding, reranking, has a dozen viable techniques, and the best one is dataset, latency, and cost dependent. Tutorials teach one trick in isolation, so teams ship a naive RAG, watch it fail, and cannot tell whether the fix is a better chunker, a reranker, a different index, or a different query strategy.
Approach
A config-driven pipeline contract (chunk, embed, index, retrieve, rerank, assemble, generate) where every stage is a swappable component behind a fixed interface, plus a runner and a leaderboard that scores any config on retrieval quality, answer quality, latency, cost, and memory. Seven phases are built so far: ingestion with deps-free parsers and from-scratch MinHash near-dup detection, 7 chunkers, tfidf and hashing embedders with int8 and binary quantization and Matryoshka truncation, from-scratch IVF and Okapi BM25 alongside optional HNSW against exact Flat, dense, sparse, hybrid, RRF, weighted and MMR retrieval, and PRF, multi-query and HyDE query transforms. The core installs on numpy and pyyaml, runs offline on an M1, and the metrics module is unit-tested against hand-computed cases. Real answer quality and cost come from Claude Haiku 4.5 on AWS Bedrock at temperature 0.
Impact
The meta-finding across phases 5 to 9 is the one that generalizes: every technique only helps where there is headroom. The same reranker lifts a weak first stage from 0.55 to 0.95 recall@1 and buys literally zero on a saturated one, at 2.7x the latency either way, and hybrid, PRF and contextual retrieval all repeat the pattern. So 'should I add X?' is unanswerable without measuring your own bottleneck first. Individual traps fell out along the way: binary quantization ships a 32x smaller index and collapses recall@1 from 0.95 to 0.05 while int8 is lossless at 4x less memory; PRF expansion hurts a corpus that already retrieves well. On real Claude Haiku, prompt style turned out to be the best return in the lab, an abstain instruction buying +17% token-F1 at identical cost, while exact match scored 0.000 on every real config and would have called the whole system broken. Phases 11 to 16 are pending.
Decisions & tradeoffs
From-scratch over library calls
MinHash near-dup, the IVF k-means coarse quantizer, Okapi BM25, and all of recall, precision, MRR, NDCG and MAP are written from the definitions and unit-tested against hand-computed cases. The core install is numpy and pyyaml with no downloads, so the mechanism stays legible and every run is reproducible offline on an 8 GB M1.
Rebuilt the eval corpus when it stopped separating configs
The phase 0 mini set saturated at recall@5 1.000, so every chunker and index scored identically and the leaderboard proved nothing. Phase 2 added a 13-doc multi-paragraph corpus with doc-level qrels and doc-level scoring, which is what made recursive at MRR 0.917 separate from fixed at 0.750.
Score retrieval phases on retrieval metrics, not answer metrics
token_f1 stayed flat at 0.21 to 0.23 across all seven chunkers because the extractive mock generator cannot convert better retrieval into a better answer. Real Claude Haiku only moved it from 0.33 to 0.38 against a one-line extractive baseline. That lexical-metric blind spot is why the LLM judge is scoped as its own later phase rather than bolted on early.
Build spec
- Status
- 11 of 17 phases · 0 through 10
- Meta-finding
- Upgrades only pay where there is headroom
- Corpus
- 13 docs, 20 labeled queries
- Tests
- 56 passing
- Generator
- Claude Haiku 4.5, Bedrock, temp 0
System notes
- Binary quantization is a 32x smaller index that collapses recall@1 from 0.95 to 0.05, int8 is lossless at 4x less memory
- PRF expansion drops recall@1 from 0.95 to 0.90 at 2x latency on an already-saturated corpus, query drift and not a free upgrade
- IVF at nprobe 1 is the ANN Pareto in one row, 2.5 points of recall@5 for 2x the speed against exact Flat
- From-scratch MinHash dedup, IVF k-means, Okapi BM25, and every retrieval metric, core install is numpy plus pyyaml
What this does not show
- The corpus is 13 documents over 20 queries, so nothing under roughly 20 points is resolvable and every number is directional.
- The harness overturned two of its own headlines: the Phase 11 judge killed the +17% prompt-style claim as a lexical artifact, and Phase 13's bootstrap put a confidence interval through the +10 point contextual-retrieval result.
Stack
Python · NumPy · BM25 · AWS Bedrock · Claude Haiku · RAG