Agentic Filing AnalystThe Integrator
An agent that answers multi-step analytical questions over SEC filings, with retrieval citations, a golden-eval CI gate, per-request tracing, guardrails, and live cost economics in one system.
Headline results
- Golden eval
- 10 of 10 PASS
- Citation recall
- 1.000
- Judge answer score
- 0.985
- Real agent cost
- $0.0737
System architecture
Problem
Ask a real analyst question, how did NVDA's data-center revenue concentration change between 2023 and 2025 and what risks does management flag, and one RAG call fails at it. The answer lives across three filings, needs arithmetic between periods, and is worthless without citations you can check. Most demos solve one slice: an agent, or retrieval, or evals. Production needs all of them plus the operational layer nobody demos.
Approach
A Claude tool-use agent at temperature 0 with four tools: hybrid BM25 and dense search with RRF fusion, exact section fetch for precise citation, cross-filing period comparison, and deterministic arithmetic so the model never does the maths itself. Guardrails screen the question on the way in and the retrieved text on the way out, because retrieved documents are an injection surface. Every request writes a SQLite trace with spans, tokens, time to first token, and dollar cost, and a golden eval set gates merges in CI on mean citation score.
Impact
On a live AWS Bedrock run with Claude Haiku 4.5, all 10 golden examples pass with mean citation recall of 1.000 and a real LLM-judge answer score of 0.985, for $0.0737 of agent-loop spend. The point is not the score, it is that one system carries the agent, hybrid retrieval, the eval gate, tracing, guardrails, and per-query cost economics, which is the shape production actually takes.
Decisions & tradeoffs
Give the model a calculator instead of trusting it with one
Percentage change and revenue share are computed by a deterministic tool the agent calls. Language models are unreliable at multi-step arithmetic and reliably confident about it, so the number never comes out of the model.
Screen the retrieved text, not just the question
An injection screen on user input is the obvious half. The other half is that anything retrieved is attacker-influenceable in the general case, so retrieved passages get their own gate before they reach the model.
Keep mock and real numbers in separate tables
Mock-mode cost and latency come from the repo's own cost model and a scripted TTFT, not from a provider. They are labelled as such and never mixed with the Bedrock run, because a portfolio number that quietly came from a mock is worse than no number.
Build spec
- Agent
- Claude Haiku 4.5 on Bedrock, temperature 0, 4 tools
- Retrieval
- Hybrid BM25 + dense, RRF fusion
- Golden eval
- 10 examples, 10/10 PASS, gate at mean ≥ 0.8
- Citation recall
- 1.000 · LLM-judge answer 0.985
- Real cost
- $0.0737 agent-loop spend
System notes
- Arithmetic is a deterministic tool, not a model output, so period comparisons cannot drift
- The retrieved text passes a second injection screen: a filing is untrusted input, not context
- Every request traces spans, tokens, TTFT and cost, so the economics are measured rather than estimated
- Mock mode is byte-identical across runs and key-free, which keeps CI honest and free
What this does not show
- Ten golden questions is a gate, not a benchmark; the citation recall of 1.000 says citations resolve, not that the analysis is right.
- Mock-mode cost and latency come from the repo's own cost model, never from a provider, and are reported separately.
Stack
Claude on Bedrock · Hybrid RAG · Golden eval + CI · SQLite tracing · Guardrails · FastAPI