No. 452025Recommender systems · Offline evaluation

Two-Stage RecommenderRetrieval Then Ranking

A production-shaped recommender: matrix-factorization candidate retrieval feeding a LightGBM ranker, evaluated leave-last-out on Recall@k, NDCG@k and MAP@k against a popularity baseline.

SVD retrieval: Recall@10 0.1405
Popularity baseline: 0.0745
Lift: ≈1.9×
Two-stage: 0.1065, below retrieval alone

System architecture

Problem

Real recommenders are two-stage because scoring millions of items with a heavy model is impossible: retrieval narrows to a few hundred cheaply, ranking spends the expensive model only on those. Most portfolio recommenders stop at cosine similarity top-k, which skips both the architecture and the question of what each stage contributes.

Approach

Stage one retrieves with truncated-SVD user and item embeddings, scoring all items and keeping the top N. Stage two re-ranks those candidates with a LightGBM classifier over richer features: the SVD score, item popularity, category affinity, user activity, and item quality. Evaluation is leave-last-out, holding out each user's most recent interaction so nothing leaks into training, scored on Recall@k, NDCG@k and MAP@k against a popularity baseline so each stage's contribution is explicit.

Impact

Both personalized systems beat the popularity baseline decisively, with SVD retrieval reaching Recall@10 of 0.1405 against 0.0745, roughly 1.9×. The honest complication is that the LightGBM ranker scores below retrieval alone at 0.1065, so on this synthetic data the second stage cost recall rather than adding precision. That is reported rather than tuned away, because a ranker that does not beat its own candidate generator is the normal first result and the reason ranking features get iterated on.

Decisions & tradeoffs

Report the stage that did not help

The LightGBM ranker scores below SVD retrieval alone on this data. Publishing that is more useful than tuning until the architecture looks right, because it identifies the ranking features as the thing to work on.

Hold out the most recent interaction, not a random one

Random held-out positives let a model use a user's later behaviour to predict their earlier behaviour. Leave-last-out matches how the system would actually be used.

Keep retrieval cheap on purpose

SVD embeddings score every item in the catalogue quickly, which is the whole reason the first stage exists. Anything heavier there defeats the two-stage architecture regardless of how good the ranker is.

Build spec

Evaluation: Leave-last-out, 2,000 test users, k = 10
popularity: Recall 0.0745 · NDCG 0.0375 · MAP 0.0264
svd retrieval: Recall 0.1405 · NDCG 0.0690 · ~1.9× baseline
two_stage: Recall 0.1065 · NDCG 0.0502
Stack: Truncated-SVD → LightGBM ranker → /recommend API

System notes

Leave-last-out evaluation holds out each user's most recent interaction, so nothing leaks
Each stage is scored separately against the baseline, which is what makes the second stage's regression visible
The popularity baseline is not a formality: it is the bar personalization has to clear and often does not
Recall, NDCG and MAP together, because ranking quality and retrieval coverage are different questions