Embedding & Reranker BenchWhich Retriever, Measured
Five embedding models and a cross-encoder reranker measured on one labeled, fully fictional retrieval set, with the reranker's contribution isolated for each embedder.
Headline results
- After rerank
- Every model at R@1 1.000
- Weakest raw
- 0.800 → 1.000
- e5-small vs mpnet
- 33M beats 109M
- Corpus
- Fully fictional
System architecture
Problem
Picking an embedding model for a RAG system is usually done by leaderboard position or parameter count. Neither predicts performance on your corpus, and both ignore the question that usually matters more: whether a cheap embedder plus a reranker beats an expensive embedder alone.
Approach
Five sentence-transformer models and a cross-encoder reranker over one labelled retrieval set, on a real Apple M1 run. The corpus is entirely fictional, so nothing can be answered from pretraining and a query only scores if the embedder matched meaning to the right passage. Each model is measured on recall@1, recall@5, MRR@10 and NDCG@10 across 30 labelled queries, broken down by query kind, alongside encode throughput, per-query latency, dimension and parameter count. Then every model's top ten is re-scored by the cross-encoder so the reranker's contribution is isolated per embedder.
Impact
Reranking lifted every embedder to a perfect recall@1 of 1.000, including the weakest raw retriever at 0.800. So a cheap fast bi-encoder plus a reranker beats a bigger bi-encoder used alone, and two cheap stages beat one expensive one. Size did not predict quality either: e5-small at 33M parameters beat mpnet at 109M on every quality metric and encoded faster. The honest mechanism is that recall@5 was already near 1.0, so the reranker's job was purely to push the answer to rank 1.
Decisions & tradeoffs
Write the corpus from nothing
Public retrieval sets leak into pretraining, and a model that has memorised the answer scores well without retrieving anything. A fictional corpus is the only way to be sure the embedding is doing the work.
Isolate the reranker per embedder
Reporting one reranked number hides whether the reranker rescues weak retrieval or just polishes strong retrieval. Running it against all five shows it does the former, which changes the buying decision.
Explain why the perfect scores are believable
Every model hitting recall@1 1.000 looks like a broken benchmark. It is not: recall@5 was already near 1.0, so the reranker only had to reorder. Saying that plainly is what keeps the result usable.
Build spec
- Setup
- 30 labeled queries over 36 passages, real M1 run
- minilm
- R@1 0.800 → 1.000 with rerank, 22.7M params
- e5-small
- R@1 0.933 → 1.000, 33.4M, beats mpnet on all metrics
- mpnet
- R@1 0.833 → 1.000, 109.5M params, 114 passages/s
- After rerank
- Every model reaches R@1 1.000
System notes
- A fictional corpus means no query can be answered from a model's pretraining
- The reranker's lift is measured per embedder rather than once, which is what shows it substitutes for model size
- e5-small (33M) beats mpnet (109M) on every quality metric and on throughput
- Recall, MRR and NDCG are implemented from scratch and unit-tested against hand-computed cases
What this does not show
- Recall@5 was already near 1.0, so the reranker only had to reorder. That is why every model reaches a perfect R@1 and why the result would not transfer to a corpus with weak recall.
Stack
sentence-transformers · Cross-encoder · MiniLM / MPNet / BGE / E5 / GTE · From-scratch metrics