Feature StoreThe Leak You Can Measure
A minimal but correct feature store with point-in-time joins offline and a SQLite online store, built to make the leakage it prevents measurable rather than asserted.
Headline results
- Naive join
- 1.00 offline → 0.61 online
- Train/serve skew
- 0.39
- Correct join
- 0.74 = 0.74
- Method
- merge_asof, point-in-time
System architecture
Problem
A training row joined against the latest value of a feature has seen the future. The model scores beautifully offline and then underperforms in production, and because both numbers are real nobody can tell which one is lying. Feature stores are usually sold as infrastructure; the reason they exist is this one failure, and almost nobody measures it.
Approach
Two paths over the same event data. The offline path builds training rows with a point-in-time-correct as-of join, a backward merge_asof keyed on the label timestamp, so a feature event dated after the label can never enter the row. The online path materialises the latest value per entity into SQLite and serves it. A churn dataset is constructed so the trap is explicit: the feature carries a weak genuine pre-label signal and a strong post-label one, so a naive join looks great and a correct one does not.
Impact
The naive latest-value join scores a perfect 1.00 offline and 0.61 in production, which is worse than the honest model, for 0.39 of train-serve skew. The point-in-time join scores 0.74 offline and 0.74 online. That table is the entire argument for a feature store in two rows: the offline number you celebrate should be the online number you ship.
Decisions & tradeoffs
Engineer the dataset so the trap is unavoidable
The feature carries a weak real signal before the label and a strong one after it. That makes the leaky join score perfectly and the correct join score modestly, which is precisely the situation where a team ships the wrong one.
Keep the wrong join in the API
get_historical_features takes a leaky flag. Deleting the broken path would make the repo tidier and remove the only thing that proves the correct path matters.
Stay minimal on purpose
No registry service, no orchestration, no vendor. Point-in-time correctness and an online store are the two ideas that carry the value, and wrapping them in infrastructure would bury the measurement this project exists to show.
Build spec
- Offline join
- pandas merge_asof, backward, point-in-time correct
- Online store
- SQLite, latest materialised value per entity
- Naive join
- 1.00 offline → 0.61 online (0.39 skew)
- Correct join
- 0.74 offline = 0.74 online
- Run
- One command, self-contained
System notes
- merge_asof with backward direction on the label timestamp, so no feature event from the future can enter a training row
- The leaky join is kept as a first-class option, because the failure is the demonstration
- A perfect offline score is presented as the warning sign it is, not the result
- Online serving reads from materialised SQLite, so the two paths are genuinely separate code paths
Stack
pandas · merge_asof · SQLite · scikit-learn · Train/serve skew