No. 102026ML infrastructure · Load-tested over real HTTP

Model Serving PlatformMeasured Batching, Safe Rollout, Overload Behavior

A production-style model inference platform with a versioned registry, dynamic micro-batching, canary and shadow deployments, and load shedding, each mechanism measured over real HTTP rather than asserted.

Throughput: 8.0× (121 → 966 RPS)
p99: 599ms → 163ms
Canary: 10% configured, 11% observed
Shed: 562 requests, bounded p99

System architecture

Problem

Most ML portfolios stop at the model working and skip the serving-infrastructure layer that decides whether it survives production traffic. Batching under load, safely rolling out a new version, and staying alive under overload are usually skipped or asserted without measurement. This project builds that layer for real and load-tests it over real HTTP to show what each mechanism is actually worth.

Approach

A thread-safe versioned registry serves models under stable and candidate aliases, shipping two real trained digit classifiers (v1 RandomForest-300 stable, v2 HistGradientBoosting candidate) so version comparisons are meaningful. A dynamic micro-batcher lands requests on a bounded asyncio queue, greedily drains what is already queued (Triton-style adaptive batching), and runs one vectorized predict per batch via a thread pool. The same queue does load shedding: past max_queue_depth, new requests get an immediate 429 plus Retry-After. A router adds canary (a configured fraction of traffic to the candidate, metered per version), shadow (candidate mirrored and compared but never returned), and instant rollback by repointing the stable alias. A FastAPI service exposes predict, deploy, status, and stdlib Prometheus-format metrics, with a load-test harness firing real HTTP requests to measure all four mechanisms end to end.

Impact

Over real HTTP (800 requests at concurrency 64), dynamic batching delivered 8.0x throughput: 121 to 966 RPS with p99 dropping 599ms to 163ms at a mean batch of 30.8. Canary split configured 10% against observed 11.0%, shadow logged 3.0% disagreement across 400 mirrored calls, and overload (queue capped at 8) admitted 238 requests at p99 296ms while shedding 562 with fast 429s. The honest arc: batching first measured 0.45x (worse than none) until the payload was made realistic and the windowing switched to adaptive drain, taking it from 0.45x to 8.0x on the same code path.

Decisions & tradeoffs

Adaptive batching over naive windowing

The first batcher awaited the full window on every batch, which added latency even under load. Switching to Triton-style greedy drain (wait only when the batch is still small) is what turned a 0.45x regression into an 8.0x gain.

Real trained payloads, not mocks

A LogisticRegression payload was too cheap for batching to amortize anything, so the stable model was swapped to a RandomForest-300 with real per-call cost. This made both the batching gain and the version comparisons meaningful.

Shadow before promote

Shadow deployment serves 100% of users from stable while mirroring traffic to the candidate and comparing answers. The 3.0% disagreement it surfaces is the exact number used to decide whether the candidate is safe to promote at zero user risk.

Build spec

Throughput gain: 8.0x (121 to 966 RPS)
p99 latency: 599ms to 163ms with batching
Shadow disagreement: 3.0% over 400 mirrored calls
Tests: 7 pytest (in-process ASGI)
Stack: FastAPI, uvicorn, asyncio, scikit-learn

System notes

Dynamic micro-batching measured at 8.0x throughput (121 to 966 RPS, p99 599 to 163ms)
Canary and shadow deployments with instant rollback by repointing the stable alias
Load shedding returns 429 plus Retry-After once the bounded queue is full
Honest finding: batching was 0.45x until the payload and windowing were fixed