Realtime Decision IntelligenceStream To Remediation
An AIOps system that ingests a durable telemetry stream, classifies the incident, quantifies its own uncertainty, and picks the remediation worth taking, end to end on a laptop.
Headline results
- Events lost on crash
- 0 of 1,200
- macro-F1
- 0.863 → 0.968
- Policy value
- 245.1 vs 206.3
- Hot path p99
- 0.919ms
System architecture
Problem
An incident-detection model that is right 86% of the time still leaves an operator asking the only question that matters: do I act on this one, and with which remediation? Detection and decision are usually separate systems, so the accuracy number never turns into an action, and the ~5% of events whose metrics genuinely look identical get guessed at.
Approach
One pipeline, six layers, on synthetic telemetry generated with labelled incidents so every claim can be checked against ground truth. A durable append-only log implements the Redis Streams contract with consumer groups and a pending ledger. Windowed features are computed by the same code offline and online. A classifier labels the incident, conformal prediction attaches a set with a coverage guarantee, an LLM reads the log line for the events metrics cannot separate, and a cost-aware T-learner estimates what each remediation is actually worth before choosing one. PSI monitors drift on top.
Impact
Nothing is lost when a consumer is killed mid-stream, 0 of 1,200 events. Train and serve features match at 0.0 skew. The classifier reaches macro-F1 0.863 on metrics alone and 0.968 once the log line is read, with dependency-failure recall going from 0.50 to 0.96. The T-learner policy scores 245.1 against 206.3 for the best risk-threshold baseline while intervening on 21% fewer events. Hot path p99 is 0.919ms at 2,513 events per second on one core. 187 tests.
Decisions & tradeoffs
Cut the MLX encoder on evidence
A learned temporal encoder produced 5× the features and made the target class worse, dependency-failure recall dropping from 0.50 to 0.41. It was removed. Keeping a component because it is the interesting part of the build is how systems get slower and less accurate at once.
Condition uplift on features, not on class
A per-class remediation table scored 108.6. The same policy conditioned on the event's features scored 245.1. The granularity of the decision mattered more than the model behind it, which is not what the intuition says.
Report that conformal cannot save you from confidence
Conformal sets hedge uncertainty, and a model that is confidently wrong is not uncertain. One failure case sits at P=0.973 on the wrong class and the set contains only that class. The guarantee is marginal coverage, not per-instance correctness, and the repo pins that as a test.
Build spec
- Ingest
- Durable log, at-least-once, 0 of 1,200 lost on crash
- Features
- Train = serve, 0.0 skew
- Classifier
- macro-F1 0.863 metrics-only → 0.968 with log reading
- Policy
- Cost-aware T-learner, value 245.1 vs 206.3 baseline
- Hot path
- p99 0.919ms · 2,513 events/s · 187 tests
System notes
- Conformal sets hold their promised coverage: 0.962 measured against a 95% target
- Reading one log line for the ambiguous ~5% of events is worth more than any model change tried on the metrics alone
- Drift monitoring alerts on 100% of drifted windows and only 8% of clean ones, after naive PSI proved unable to tell an incident from drift
- The anomaly detector had to leave the hot path and then leave the thread, because co-location inflated p99 by 2.4× with work nothing was counting
What this does not show
- The telemetry is synthetic, generated with labelled incidents so every claim can be checked against ground truth.
- The live-model log reader is written but unrun (expired credentials), so the log-reading numbers come from a deterministic offline reader.
Stack
Streaming ingest · scikit-learn / LightGBM · Conformal prediction · T-learner · Claude on Bedrock · Next.js