Self-Improving Agent PlatformThe Closed Retraining Loop
A multi-tenant RAG-agent platform that measurably improves itself with no human in the retrain loop. Every trace is scored, failures are mined into eval cases, a router is retrained on-device, shadow-tested, and promoted only on measured lift.
Headline results
- Cost per query
- 2.6¢ → 1.1¢
- Quality held
- 92% → 92%
- Unattended weeks
- 6 · $1.77
- Tests
- 245
System architecture
Problem
Every LLM agent in production degrades or overspends, and the fix is always a human: someone reads the traces, notices the pattern, rewrites the prompt, swaps the model. That human is the bottleneck and the reason most agent projects stall. Nobody ships the loop that does it without them, because closing it means solving the hard half first, an objective quality signal you can promote on.
Approach
A multi-tenant platform in three planes. Serve is a grounded plan, retrieve, tool, critic agent on real AWS Bedrock whose inline citations are checked against what retrieval actually returned, with a sandboxed DuckDB tool it uses to run its own SQL before answering. Observe traces every request to SQLite and scores it two ways: an execution oracle that checks SQL answers by running them, and an LLM judge calibrated against that oracle rather than trusted on its own. Self-improve mines low-scoring traces into training data, retrains the router on-device, replays it in shadow for free, gates it behind an execution-oracle canary, and promotes only on measured dominance. A promotion log records every rejection, and rollback is one command.
Impact
Six unattended simulated weeks of support traffic on real Bedrock for $1.77 total: quality held at 92% while cost fell from 2.6¢ to 1.1¢ per query, a 54% reduction with no human in the retrain loop. The flywheel rejected its candidate twice for insufficient evidence before promoting at week 3, and the second rejection is the interesting one, because it triggered the bounded shadow sample that produced the missing evidence. Then it declined to churn when nothing further could be gained. 245 offline tests, a FastAPI backend, a Next.js chat and admin console, and a docker compose stack.
Decisions & tradeoffs
Pick a corpus whose answers are executable
DuckDB documentation won over SEC filings and FastAPI docs for one reason: SQL answers can be verified by running them. A well-memorized corpus lets the model answer from parametric memory and mask retrieval failure, and a judge-only signal is exactly what a self-improving loop learns to game.
Log the rejections, not just the promotions
Cycle 1 was rejected for insufficient evidence and that rejection ships in the promotion log. A flywheel that only records its wins is marketing. Keeping the rejections is what makes the promotion at week 3 mean something.
Call the first promotion what it is, a demotion
The policy that won was always-cheap, a declared-degenerate router that kills the waste the agent milestone had measured by hand. Dressing that up as learned intelligence would be dishonest; the honest read is that the loop found the cheapest correct answer and the escalation net still catches the rest.
Build spec
- Corpus
- DuckDB docs, 411 pages, pinned by commit sha
- Retrieval
- Dense + link graph, R@1 0.371 · R@10 0.843 · MRR 0.573 at ~11ms
- Models
- Claude Haiku 4.5 cheap tier · Sonnet 4.6 strong tier, on Bedrock
- The curve
- 6 weeks · quality 92% → 92% · cost 2.6¢ → 1.1¢ · $1.77
- Tests
- 245 offline · FastAPI + Next.js + docker compose
System notes
- Execution-based oracle: SQL answers are scored by running them, not by asking another model, which is what stops the loop reward-hacking its own judge
- The CI gate flips 92% green to 67% red on a deliberately worse prompt, and it caught a citation collapse an execution-only gate would have missed
- The on-device reranker fine-tune was rejected correctly: it lifted recall@3, recall@10 and nDCG but regressed rank 1 from 0.357 to 0.314, so the dominance gate declined it
- Every guardrail fired on a live run, including an injection blocked at $0.00 before it reached the model
What this does not show
- Traffic is authored: 12 queries a week at ±8% granularity, so the curve demonstrates the loop rather than a trend.
- One promotion event is not a trend, and the policy that won is honestly degenerate (always-cheap).
- The golden eval is 12 cases with a starting-guess threshold; post-promotion grounding was 87.5% against 93.8% pre, which n=24 cannot resolve from noise.
Stack
AWS Bedrock · FAISS + BM25 · FastAPI · Next.js · MLX / torch on-device · SQLite tracing