Demand ForecastingBacktested Honestly
A productionized demand forecasting system: naive baselines through Holt-Winters and LightGBM, scored by rolling-origin backtest across folds and SKUs, served behind an API with prediction intervals.
Headline results
- Holt-Winters
- MAPE 9.18%, best
- LightGBM
- MAPE 9.25%
- Naive baseline
- MAPE 19.85%
- Backtest
- Rolling origin, 4 folds
System architecture
Problem
Forecasting is where evaluation goes wrong quietly. A random train-test split lets the model see the future, every metric looks excellent, and the error only shows up in the operational plan the forecast fed. The separating skill is not the model class, it is refusing to test on the past.
Approach
Five model classes over multi-SKU data: naive, seasonal naive, moving average, Holt-Winters implemented from scratch, and LightGBM with lag, rolling and calendar features doing recursive multi-step forecasts. All scored by rolling-origin backtest, fit then forecast forward, across four folds and 32 evaluations, on MAPE, sMAPE, MAE and RMSE. Nothing counts until it clears the naive baseline. The winner is served behind a forecast API with prediction intervals.
Impact
Holt-Winters and LightGBM roughly halve the naive baseline's error, 9.18% and 9.25% MAPE against 19.85%, and the backtest proves it on held-out future windows rather than on a shuffled split. The from-scratch Holt-Winters narrowly beats LightGBM, which is worth stating: on clean seasonal data, gradient boosting bought nothing over a classical method with the right structure.
Decisions & tradeoffs
Rolling origin, never a random split
A shuffled split on time-series data leaks the future into training and produces a number that cannot be reproduced in operation. Every model here is fit on the past and scored forward, which is the only evaluation that transfers.
Implement Holt-Winters rather than import it
The level, trend and seasonal update equations are the model. Writing them makes the smoothing parameters inspectable and gives the tests something concrete to check, and it turned out to be the winning model.
Ship prediction intervals, not point forecasts
An inventory decision needs a range. A single number invites treating the forecast as certain, which is how a good model produces a bad plan.
Build spec
- Backtest
- Rolling-origin, horizon 14, 4 folds, 32 evaluations
- holt_winters
- MAPE 9.18% · sMAPE 8.84% · best
- lightgbm
- MAPE 9.25% · sMAPE 8.89%
- seasonal_naive
- MAPE 11.12%
- naive
- MAPE 19.85%, the bar
System notes
- Rolling-origin backtest across 4 folds and 32 evaluations, never testing on the past
- Holt-Winters written from scratch, so the seasonal decomposition is auditable rather than a library call
- LightGBM forecasts recursively for multi-step horizons, which is the honest way to score a 14-day forecast
- The naive baseline is reported in every table as the bar everything must clear
Stack
LightGBM · From-scratch Holt-Winters · Rolling-origin backtest · FastAPI