No. 122026Backend and LLMOps · Verified against real Bedrock

LLM GatewayProduction LLM API Gateway for Platform Teams

A production LLM API gateway that puts auth, per-tenant rate limiting and token quotas, ordered provider failover, SSE streaming, cost metering, and an append-only audit log on one FastAPI request path, with the failover verified against real AWS Bedrock.

Failover: Dead primary → 200 on attempt 2
Rate limit: Token bucket, 429
Quota: Per-tenant, 402
Audit: Append-only, prompts never logged

System architecture

Problem

Every app that calls an LLM provider directly re-solves the same five concerns: who is allowed to call it, how much they can call it, what happens when the upstream is down, what each tenant costs, and what the compliance trail is. A platform team wants to solve these once, centrally, instead of having every app reinvent auth, limits, and failover. This project builds that layer as a single request path and proves the failover works against a real provider, not just a mock.

Approach

The gateway is one FastAPI request path: API-key auth, then a per-tenant token-bucket rate limiter (429 plus Retry-After), then a cumulative token quota check (402), then an ordered provider failover chain over AWS Bedrock and the Anthropic API, then per-tenant cost metering and an append-only audit log. Rate limiting and quotas are deliberately two separate mechanisms because burst pressure and total spend are different questions. Every upstream sits behind one interface returning text and usage, so adding a provider is one class and the chain itself is configuration. Streaming uses SSE passthrough and fails over only before the first chunk is emitted, since replaying a partially streamed response would duplicate or contradict output. The audit log records request id, tenant, provider, tokens, cost, latency, and status, but never the prompt.

Impact

Verified end to end against real infrastructure with a deliberately dead primary in front of real AWS Bedrock: the failed attempt was absorbed and the retry served by Bedrock on attempt 2 at 2135ms, with the real token usage metered into the tenant's bill and the whole event recorded in the audit log. The client never saw the outage. pytest passes 8 of 8 offline against stub upstreams, covering auth, rate limiting, quota, failover, all-providers-down, metering math, tenant-scoped audit, and SSE streaming.

Decisions & tradeoffs

Rate limit is not quota

The token bucket bounds burst pressure in requests per second and answers with 429 plus Retry-After, while the quota bounds total spend in cumulative tokens and answers with 402. A tenant can be well inside their rate limit and still out of budget, so production billing needs both signals.

Stream failover only before first byte

If a provider dies mid-stream the client has already received partial output, so replaying on another provider would silently duplicate or contradict it. The gateway fails over only when an upstream dies before emitting its first chunk, and a mid-stream death surfaces as an SSE error event.

Prompts are never audit-logged

The audit trail carries request id, tenant, provider, tokens, cost, latency, and status, which is enough for compliance and debugging. Leaving prompts out keeps the log from becoming a PII store.

Build spec

Providers: AWS Bedrock primary, Anthropic API secondary
Endpoints: /v1/chat, /v1/usage, /v1/health, /v1/audit, /healthz
Limits: Token-bucket rate limit + per-tenant token quota
Tests: 8/8 pytest, offline against stub upstreams
Failover demo: Attempts 2, served by Bedrock, 2135ms

System notes

Ordered failover verified against a dead primary, served by real Bedrock on attempt 2
Per-tenant token-bucket rate limiting (429) plus cumulative token quotas (402)
SSE streaming passthrough with fail-over only before the first chunk
Append-only audit log that records tokens and cost but never the prompt