Vector Index BenchmarkRecall Against Exact Truth
A reproducible ANN benchmark measuring recall against exact ground truth, latency, throughput, build time and memory across five vector indexes on one identical workload.
Headline results
- faiss_ivf
- recall 1.000 at 23,467 QPS
- faiss_flat
- recall 1.000 at 2,355 QPS
- faiss_hnsw
- 0.996 at 12,733 QPS
- faiss_ivfpq
- 3.0 MB, recall 0.726
System architecture
Problem
Every RAG and agent stack needs a vector index, and the choice trades speed against recall against memory. Vendor benchmarks measure recall against another approximate index, which makes the number meaningless, and rarely hold the workload constant across candidates.
Approach
One workload, identical vectors, queries and ground truth, run across FAISS flat, IVF, IVFPQ and HNSW plus hnswlib. Recall is measured against exact brute-force ground truth rather than against another approximate index, so the numbers mean what they say. Each index reports recall@k, latency at p50 and p95, queries per second, build time and index memory, plotted as a recall-throughput Pareto chart. The adapters share one three-method interface, so a server-backed store like Qdrant, Milvus or pgvector drops in without touching the harness.
Impact
On 50,000 vectors at 96 dimensions with k=10, FAISS IVF holds perfect recall at 23,467 queries per second, roughly ten times exact flat search at the same 1.000 recall. HNSW gives up 0.4% of recall for 5.4× the throughput of flat. IVFPQ is the memory play and the honest warning: 3.0 MB against 19.7, for recall falling to 0.726. The Pareto chart is the deliverable, because the right index depends on which of those three axes is binding.
Decisions & tradeoffs
Compute exact ground truth first
Brute-force search over the whole corpus is expensive and it is the only way a recall number is trustworthy. Every approximate index is scored against it rather than against each other.
Publish a Pareto chart instead of a winner
IVF wins on this workload at this scale. That is a fact about this workload. The chart is the honest output because the answer changes with corpus size, dimension and memory budget.
Keep the adapter interface to three methods
Build, add, search. Anything richer starts encoding one library's assumptions and stops being a fair harness for the others.
Build spec
- Workload
- 50,000 × 96-dim, 1,000 queries, k = 10
- faiss_ivf
- recall 1.000 · p50 0.041ms · 23,467 QPS · 19.7 MB
- faiss_flat
- recall 1.000 · p50 0.370ms · 2,355 QPS (exact)
- faiss_hnsw
- recall 0.996 · 12,733 QPS · 32.8 MB · 1.11s build
- faiss_ivfpq
- recall 0.726 · 3.0 MB, the memory play
System notes
- Recall is scored against exact brute-force ground truth, not against another approximate index
- One shared three-method adapter interface, so server-backed stores plug into the same harness
- Index memory is reported, which is the axis that decides whether the thing fits on the box you have
- The Pareto chart makes the trade visible rather than declaring a single winner
Stack
FAISS · hnswlib · matplotlib · Exact ground truth