Realtime ML PipelineStreaming Fraud Scoring with Proven At-Least-Once Delivery
An end-to-end streaming fraud scorer where every operational guarantee, at-least-once delivery, train/serve feature consistency, and PSI drift alerting, is demonstrated by a measurement rather than asserted.
Headline results
- Events lost on crash
- 0 of 800
- Throughput
- ~10k/s at p99 0.07ms
- Fraud
- 82% precision · 92% recall
- Drift
- Fires on drifted stream only
System architecture
Problem
Streaming ML systems fail in ways batch systems do not: a crashed consumer silently drops events, features computed one way in training and another at serving time rot the model unnoticed, and a shifting input distribution goes undetected until business metrics fall. Most streaming demos skip all three and just show a model scoring a queue. This project makes each failure mode concrete, reproducible, and measured on a fraud-scoring pipeline.
Approach
A durable, file-backed event log implements the exact Redis Streams contract (XADD/XREADGROUP/XACK/XAUTOCLAIM) with consumer groups and a per-group PENDING ledger, so delivery semantics are testable in-process. The consumer loop claims a batch, computes online per-card windowed features, scores, alerts, feeds the drift monitor, and ACKs only after processing, which is what makes delivery at-least-once. A class-weighted LogisticRegression trains on a historical replay pushed through the same OnlineFeatures class that serves live traffic, removing train/serve skew. A rolling-window PSI drift monitor alerts when live transaction amounts diverge from the training reference beyond a 0.2 threshold. Redelivered duplicates are deduped by offset for idempotent processing.
Impact
Measured on 3000 events: throughput around 10,000 events/s with per-event latency p50 0.044ms and p99 0.069ms, and fraud detection at precision 82% and recall 92% versus injected ground truth. Crash recovery is proven, not claimed: a consumer killed after 150 of 800 events left a rescue consumer to redeliver and process the remaining 650, losing 0 of 800 events with an empty pending ledger. The drift monitor fired 0 false alarms on a normal stream and 3 alerts (PSI 2.88 vs 0.2) on a drifted stream.
Decisions & tradeoffs
ACK after processing
The consumer acknowledges events only after they are fully processed, so events claimed but unacked by a crashed consumer stay in the PENDING ledger until a peer redelivers them. This ordering is precisely what guarantees at-least-once delivery.
One feature code path
The model trains on a historical replay pushed through the same OnlineFeatures class that scores live traffic. A single code path eliminates the class of bug where batch training features and streaming serving features silently diverge.
Redis Streams contract on a local log
The broker replicates the exact XADD/XREADGROUP/XACK/XAUTOCLAIM semantics on a file-backed append-only log, so CI runs the full crash-recovery and delivery suite with no external services. Swapping in real Redis is one adapter with the same five methods.
Build spec
- Throughput
- ~10,000 events/s (single consumer)
- Latency
- p50 0.044ms · p99 0.069ms
- Fraud detection
- precision 82% · recall 92%
- Crash recovery
- 0 of 800 events lost
- Stack
- Python, NumPy, scikit-learn, pytest
System notes
- 0 of 800 events lost under a mid-batch consumer crash
- Fraud detection precision 82%, recall 92%
- ~10,000 events/s at p99 0.069ms latency
- PSI drift monitor: 0 false alarms, fires only on shift
Stack
Python · Streaming · Fraud detection · At-least-once · Drift detection · scikit-learn