LLM RouterThe Cheapest Model That Can Still Answer
A router that sends each query to the cheapest model that can still answer it, benchmarked against always-small and always-large on accuracy and real per-token cost across both a deterministic mock and a live Bedrock run.
Headline results
- Learned router, real
- −29% cost at 0.917 quality
- Heuristic, mock
- 96% quality at 51% cost
- The finding
- Mock and real invert
- Cost model
- Real per-token
System architecture
Problem
Most production LLM traffic is easy. Greetings, lookups, and formatting do not need a frontier model, yet teams pay top-model prices for every token. Routing is the obvious answer and the unmeasured one, because without difficulty labels there is no way to say what a router costs you in quality.
Approach
A query set is tagged easy or hard, where hard means the cheap model actually gets it wrong, which is what makes routing measurable at all. Two routers compete: a transparent heuristic with no training that sends long prompts and code, math, or multi-step reasoning cues to the large model, and a learned TF-IDF plus logistic-regression router predicting P(hard) and routing on a threshold. Both run against always-small and always-large baselines over the same 24 held-out queries, reporting accuracy, total cost from real per-token pricing, and cost saved versus always-large at matched quality. The provider auto-selects AWS Bedrock, then the first-party Anthropic API, then a deterministic mock, so reviewers reproduce every mock number at zero cost while the real run grades Claude Haiku 4.5 against Claude Opus 4.6 with Opus as judge.
Impact
Which router wins flips between the mock and the real model, which is the finding. On the mock the transparent heuristic reaches 96% of always-large quality at 51% of the cost and beats the learned router by 12 points at the same 46% escalation rate, because difficulty there is structural rather than lexical and TF-IDF overfits vocabulary. On real Bedrock the ranking inverts: the learned router matches always-large quality at 0.917 for 71% of the cost, a 29% saving for zero quality loss, because it adapts to where real Haiku actually fails. Real Haiku is also far stronger than the mock's worst case at 0.833 always-small, which is precisely why routing rather than always-large is the right default. The benchmark ships both routers and computes the recommendation rather than hard-coding it.
Decisions & tradeoffs
Label difficulty, or routing is unmeasurable
Hard is defined as a query the cheap model gets wrong, not a query that looks hard. Without that label there is no way to separate a router that saves money from one that quietly degrades answers.
Keep the losing router in the benchmark
The learned router loses on the mock by 12 points and wins on real Bedrock by 29% cost at matched quality. Shipping both is what surfaced the flip, and it is a cleaner result than either router alone.
Deterministic mock plus a real run
The mock needs no keys so CI and reviewers reproduce every number for free, and it deliberately models a weaker small model to make the routing logic visible. The Bedrock run confirms the payoff against real models and reports where the two disagree.
Build spec
- Models
- Claude Haiku 4.5 / Opus 4.6 (Bedrock)
- Eval set
- 24 held-out queries, easy/hard labels
- Real learned router
- 0.917 acc, 71% of always-large cost
- Mock heuristic
- 0.958 acc, 51% of always-large cost
- Stack
- scikit-learn, anthropic[bedrock], pytest
System notes
- Real Bedrock: learned router matches always-large 0.917 at 71% of cost, a 29% saving
- Mock: heuristic hits 96% of large-model quality at 51% of cost, +12 pts over learned
- Which router wins flips between mock and real, so the recommendation is computed
- Full real run graded by Opus as judge over 24 held-out queries for about $0.22
What this does not show
- Do not quote the mock's heuristic-wins result as the headline. On real Bedrock the learned router wins, because it adapts to where Haiku actually fails.
Stack
Python · scikit-learn · Claude · AWS Bedrock · LLMOps · Cost