Semantic CacheThe Threshold Is the Product
A similarity cache for LLM apps benchmarked across the full threshold sweep on both sides of the trade: cost and latency saved, and the false-hit rate of serving a wrong cached answer to a near-miss.
Headline results
- Real embeddings
- 17% cost cut, zero false hits
- Lexical key
- 3% at the same safety
- Safe band
- 5.7× wider
- At a 2.8% budget
- 47% saved
System architecture
Problem
Every repeated question sent to an LLM is money paid twice. A semantic cache serves a stored answer when a new query is close enough to one already answered, but the hard part was never the cache. It is the similarity threshold, because a cache tuned too loose starts serving the wrong cached answer to a query that merely looks similar, and nobody measures that.
Approach
A 36-query workload built with two structures a cache must get right: paraphrase clusters, where several differently-worded queries share one answer and the cache should miss the first then hit the rest, and near-misses, where queries are close but have different answers so a hit serves the wrong answer. Every query carries its gold answer, so each hit is scored correct or wrong. The benchmark sweeps the cosine threshold end to end and reports hit rate, cost saved, latency saved, and false-hit rate at every setting, then recommends the threshold that maximizes savings inside a stated false-hit budget. Two embedders run behind the same SemanticCache interface, a deterministic word TF-IDF lexical key and real sentence embeddings via all-MiniLM-L6-v2.
Impact
Real sentence embeddings widen the safe-savings band 5.7x over a lexical key, a 17% cost cut at zero false hits versus 3%, and 47% at a 2.8% false-hit budget. The lexical key forces a brutal choice, 3% savings at zero false hits or 33% savings while serving wrong answers to 14% of traffic, because paraphrases reword more than word overlap captures while near-misses share words and trip the same low threshold. Same cache logic, same workload, only the vectors change. A semantic cache is only as safe as its embedding, and the false-hit curve is the dial you tune to your wrong-answer tolerance.
Decisions & tradeoffs
Measure the false-hit rate, not just the hit rate
A cache benchmark that reports only hit rate and cost saved is measuring one side of a two-sided trade. Every query carries a gold answer, so each hit is scored correct or wrong, which turns the wrong-answer risk into a number that moves with the threshold instead of an unstated hazard.
Sweep the threshold rather than pick one
There is no correct global threshold, only one correct for a given wrong-answer tolerance. Sweeping the full cosine range and reporting savings against false hits at each setting lets the budget choose the operating point.
Ship the lexical embedder as the contrast
The TF-IDF path is deterministic and key-free so tests and CI reproduce every number with no download. Keeping it alongside the real embedder is what makes the 5.7x band widening legible, since both run the same cache logic over the same workload and only the vectors differ.
Build spec
- Workload
- 36 queries, paraphrase + near-miss
- Safe savings
- 16.7% at 0.0% false hits
- Budgeted savings
- 47.2% at 2.8% false hits
- Lexical ceiling
- 2.8% at 0.0% false hits
- Stack
- scikit-learn, sentence-transformers, NumPy
System notes
- Real embeddings widen the safe band 5.7x: 17% cost cut at zero false hits vs 3% lexical
- Lexical key at 0.60 buys 33% savings while answering 14% of traffic wrong
- 36-query workload of paraphrase clusters plus near-misses, every query gold-labeled
- Threshold recommendation computed from the sweep against a stated budget, not hard-coded
Stack
Python · sentence-transformers · scikit-learn · NumPy · Caching · pytest