RAG Architectures13 RAG Designs Benchmarked Side by Side
13 RAG architectures engineered as one production-grade framework and benchmarked side by side on a shared corpus and labeled eval, showing that structure and iteration, not query rephrasing, are what solve multi-hop retrieval.
Headline results
- Architectures
- 13, one corpus
- Query-transform on multi-hop
- 0%
- Models
- Real Claude on Bedrock
- Scoring
- Recall + judged accuracy
System architecture
Problem
RAG architecture debates (naive vs hybrid vs graphrag vs agentic) run on unreliable evidence because almost every benchmark tests architectures in isolation on different corpora, so comparisons are not apples to apples. Tutorial code rarely includes multi-hop questions, which is exactly where the architectures diverge. And most RAG repos are notebook-grade: no typed interfaces, no retries, no tracing, no offline tests, so nothing transfers to production.
Approach
13 paper-faithful architecture packages (naive, sparse, hybrid, rerank, multi_query, rag_fusion, HyDE, chunking, GraphRAG, RAPTOR, corrective/CRAG, adaptive, agentic ReAct) run over one shared core framework with typed domain models, a dependency-injected composition root, pluggable Bedrock/Anthropic and FAISS/BM25 backends, retry plus caching, span tracing, and a StructuredCaller repair loop for JSON output. Every architecture runs over the same corpus, embeddings, and LLM under a strict fairness contract (one shared generator prompt, one shared judge, shared offline artifacts) so score differences are architecture differences, not infrastructure differences. The eval is a fully fictional interlinked knowledge base of 14 documents plus 12 labeled questions (8 single-hop, 4 multi-hop) with gold doc IDs, fictional on purpose so answers cannot be in training data and a method only scores if it actually retrieves. There are no cross-architecture imports, so each package is independently readable and deletable, and the whole framework runs offline in 43 pytest tests with no keys, network, or model downloads.
Impact
On one real Bedrock run (Claude Haiku 4.5, 12 questions, 14 docs), multi-hop accuracy was the discriminator: query-transform methods (multi_query, rag_fusion, HyDE) and the naive baseline scored 0% on multi-hop, because no rephrasing of the question resembles the bridge document. Methods that add a second mechanism reached 50% multi-hop: agentic and raptor led at 83% answer accuracy, while GraphRAG underperformed its promise at 67% (gated by extraction quality and the 2-hop limit). The honest read: with 12 questions and an LLM judge, one flipped question moves a method about 8 points, so the robust signal is structural, structure and iteration beat vocabulary breadth, not the exact rank order.
Decisions & tradeoffs
One shared runtime, not 13 setups
Every architecture receives the same injected Runtime (embedder, LLM, offline artifacts) under a fairness contract. This guarantees score differences are architecture differences rather than infrastructure differences.
A fictional knowledge base
The 14-document corpus is entirely invented so answers cannot exist in any model's training data. A method only scores if it actually retrieves the right docs, which measures RAG rather than memorization.
No cross-architecture imports
Corrective builds its own hybrid fallback and adaptive its own iterative chain from core primitives, never importing sibling packages. Each package stays independently readable and deletable at the cost of some duplication.
Build spec
- Architectures
- 13 packages: naive, sparse, hybrid, rerank, multi_query, rag_fusion, HyDE, chunking, GraphRAG, RAPTOR, corrective, adaptive, agentic
- Retrieval
- FAISS dense + rank-bm25 sparse, RRF and weighted fusion, cross-encoder rerank
- Generation
- Claude on AWS Bedrock plus direct Anthropic API, StructuredCaller repair loop
- Eval
- Fictional 14-doc KB, 12 labeled questions, recall@k, hit-rate, MRR, NDCG, LLM judge
- Tests
- 43 offline pytest tests via FakeLLM plus HashingEmbedder, zero network
System notes
- 13 paper-faithful architectures on one shared core, one corpus, one eval
- Query-transform methods score 0% on multi-hop; agentic and raptor reach 83%
- Fictional 14-doc KB so scores measure retrieval, not memorization
- Entire framework runs offline in 43 pytest tests, no keys or network
What this does not show
- A 14-document demo corpus makes the numbers directional; the ordering between architectures is the result, not the absolute scores.
Stack
Python · AWS Bedrock · FAISS · BM25 · GraphRAG · sentence-transformers