Recommendation Decision Lab: When Does Personalization Earn a Route?
Abstract
Recommendation experiments can look persuasive when they score only items a user has already encountered, yet fail when asked to retrieve from a full catalog. This project evaluates a train-only co-review recommender against a validation-selected popularity baseline using temporal splits of Amazon Reviews'23. On 35,905 eligible Musical Instruments review requests in a held-out category, the hybrid reached 0.008681 NDCG@10 versus 0.008007 for recent popularity, a paired difference of 0.000674 with a user-cluster 95% interval of [0.000040, 0.001338]. A later two-tower neural extension improved on popularity but did not beat the hybrid, so it remained a shadow route. A local service makes the validation gate, shadow ranking, and runtime fallback inspectable. The gain is small and offline; the methods are established rather than new recommendation algorithms.
A recommender that predicts ratings well may still fail at the task users actually see: finding relevant items among thousands they have never reviewed. This project asks a narrower, testable question. Can a small personalized layer beat a strong, transparent popularity control when both must rank the entire train-known catalog, and can the system decline personalization when the evidence or runtime conditions do not support it? The preprint presents the evaluation, implementation, and limitations together.
Explore the decision
Evidence first, then routing
Inspect the held-out scores, compare an exploratory neural challenger, or trace a request through a synthetic fallback scenario. Demo controls do not change measured results.
Held-out category · full catalog
Musical Instruments
Bars start at zero and share a 0 to 0.012 NDCG@10 scale.
Eligible requests
35,905
Targets unseen in training
12,305
Paired difference
+0.000674
A ranker trained on the earlier catalog cannot retrieve a product it has never seen.
User-cluster 95% interval for the paired difference: [0.000040, 0.001338]. These are offline review-retrieval scores from one temporal split, not online engagement or causal lift.
Neural retrieval extension
A challenger held in shadow
A trained history and item tower scored the full train-known catalog. Validation selected a 40% neural blend, then compared it with the active co-review hybrid. The neural route did not earn promotion.
Validation
33,993 eligible requests · NDCG@10
Bars start at zero and share a 0 to 0.012 scale.
Gate decision: keep the hybrid active
Neural minus hybrid, paired user-cluster 95% interval: [-0.000985, 0.000611]. The interval includes zero.
The unblended tower reached 0.001411 on validation. This extension was designed after the category test period had been studied, so its test view is exploratory. Switching periods changes the displayed measurements; it does not rerun a model or alter the saved routing decision.
Synthetic titles and histories
Prior items: Atlas Drift, Comet Circuit
Popularity route active
Validation gate closed: keep the baseline active.
Baseline
popularity
- 1.The Glass Grove
- 2.Hollow Harbor
- 3.Lantern Vale
- 4.Signal Runner
- 5.Tide and Timber
Shadow challenger
personalized
- 1.Signal Runner
- 2.Orbit of Ash
- 3.Hollow Harbor
- 4.The Glass Grove
- 5.Lantern Vale
This walkthrough uses invented items and precomputed toy rankings. Gate and failure controls illustrate routing behavior; they do not run the private model or change the Amazon test results.
Evaluation design
The experiment uses the Amazon Reviews'23 5-core temporal benchmark. Training reviews with ratings of at least four stars provide item popularity counts and a cosine co-review neighborhood. Each eligible positive review in validation or test becomes a retrieval request. The ranker excludes the user's earlier reviewed items, searches the full train-known catalog, and counts a target absent from that catalog as a miss.
Lifetime and recent popularity compete on validation to select the control. The challenger blends that control with co-review scores; validation also selects its blend weight and decides whether a user-cluster bootstrap supports opening the route. This choice is saved before the test archive is read. The design avoids selecting a winning model from test outcomes, although it cannot remove the benchmark's retrospective filtering or turn reviews into exposure logs.
Held-out evidence
The selected method was tested on the previously unopened Musical Instruments category. Across 35,905 eligible requests, the hybrid reached 0.008681 NDCG@10 versus 0.008007 for recent popularity. The paired hybrid-minus-popularity difference was +0.000674, with a user-cluster 95% interval of [0.000040, 0.001338].
The Musical Instruments gain is positive but small, and both absolute scores are low. Of its 35,905 eligible test requests, 12,305 targeted items absent from the training catalog. Reordering known items cannot recover those targets. The experiment therefore exposes a coverage problem as well as a ranking result.
Neural challenger
An exploratory extension trains a two-tower retriever on 313,522 chronological training pairs. One tower encodes up to 20 prior positive items; the other embeds products. Training uses in-batch negatives while masking duplicate targets and known positives. The fitted model scores the full train-known catalog, then validation selects a blend with recent popularity and compares it with the already active co-review hybrid.
The standalone neural model scored 0.001411 NDCG@10 on validation. A 40% neural blend reached 0.011774, ahead of recent popularity at 0.010715 but behind the hybrid at 0.011969. Its paired interval against the hybrid crossed zero, so the gate kept the hybrid active. On the previously studied test period, the neural blend scored 0.008357 versus 0.008681 for the hybrid. This is a development comparison, not a second independent confirmation: the category's test result was already known before the neural model was designed. The interactive neural view above separates validation from that exploratory test result.
From measurement to a usable system
A local HTTP service serves the same ranking function through a hash-checked model bundle. It can also load the neural weights after source and integrity checks, returning that ranking as a shadow while the validation gate stays closed. For each request it reports the active route alongside baseline and shadow rankings; empty or unsupported histories and challenger failures return the baseline with a reason. The interactive walkthrough above uses invented users, item names, and precomputed rankings to illustrate routing. The source repository includes the synthetic demo and aggregate results; downloaded reviews and fitted weights are not distributed.
Limits and next test
This is a category transfer within one platform and one fixed time split, not evidence of online impact. The 5-core subset uses support measured over the full corpus, and a review is neither an impression nor an unbiased engagement label. The paired interval resamples users in that split; it does not capture refitting, shared-item dependence, or future traffic shifts. One compact neural retriever is a useful comparison, but it does not exhaust modern retrieval methods. A useful next test would use exposure-logged or less-filtered activity, independent time periods, and a predeclared comparison against stronger baselines.