Recommendation Decision Lab: When Does Personalization Earn a Route?
Abstract
Recommendation experiments can look persuasive when they score only items a user has already encountered, yet fail when asked to retrieve from a full catalog. This project evaluates a train-only co-review recommender against a validation-selected popularity baseline using temporal splits of Amazon Reviews'23. On 35,905 eligible Musical Instruments review requests in a held-out category, the hybrid reached 0.008681 NDCG@10 versus 0.008007 for recent popularity, a paired difference of 0.000674 with a user-cluster 95% interval of [0.000040, 0.001338]. A later two-tower neural extension improved on popularity but did not beat the hybrid, so it remained a shadow route. An exploratory wording arm lost on Musical Instruments in both lexical and encoded forms, and its lexical form also lost on Video Games. A local service makes the validation gate, shadow ranking, separate typed-request answers, and runtime fallback inspectable. The gain is small and offline; the methods are established rather than new recommendation algorithms.
A recommender that predicts ratings well may still fail at the task users actually see: finding relevant items among thousands they have never reviewed. This project asks a narrower, testable question. Can a small personalized layer beat a strong, transparent popularity control when both must rank the entire train-known catalog, and can the system decline personalization when the evidence or runtime conditions do not support it? The preprint presents the evaluation, implementation, and limitations together.
Explore the decision
Evidence first, then routing
Inspect the held-out scores, compare an exploratory neural challenger, or trace a request through a synthetic fallback scenario. Demo controls do not change measured results.
Held-out category · full catalog
Musical Instruments
Bars start at zero and share a 0 to 0.012 NDCG@10 scale.
Eligible requests
35,905
Targets unseen in training
12,305
Paired difference
+0.000674
A ranker trained on the earlier catalog cannot retrieve a product it has never seen.
User-cluster 95% interval for the paired difference: [0.000040, 0.001338]. These are offline review-retrieval scores from one temporal split, not online engagement or causal lift.
Neural retrieval extension
A challenger held in shadow
A trained history and item tower scored the full train-known catalog. Validation selected a 40% neural blend, then compared it with the active co-review hybrid. The neural route did not earn promotion.
Validation
33,993 eligible requests · NDCG@10
Bars start at zero and share a 0 to 0.012 scale.
Gate decision: keep the hybrid active
Neural minus hybrid, paired user-cluster 95% interval: [-0.000985, 0.000611]. The interval includes zero.
The unblended tower reached 0.001411 on validation. This extension was designed after the category test period had been studied, so its test view is exploratory. Switching periods changes the displayed measurements; it does not rerun a model or alter the saved routing decision.
Synthetic titles and histories
Prior items: Atlas Drift, Comet Circuit
Popularity route active
Validation gate closed: keep the baseline active.
Baseline
popularity
- 1.The Glass Grove
- 2.Hollow Harbor
- 3.Lantern Vale
- 4.Signal Runner
- 5.Tide and Timber
Shadow challenger
personalized
- 1.Signal Runner
- 2.Orbit of Ash
- 3.Hollow Harbor
- 4.The Glass Grove
- 5.Lantern Vale
This walkthrough uses invented items and precomputed toy rankings. Gate and failure controls illustrate routing behavior; they do not run the private model or change the Amazon test results.
Evaluation design
The experiment uses the Amazon Reviews'23 5-core temporal benchmark. Training reviews with ratings of at least four stars provide item popularity counts and a cosine co-review neighborhood. Each eligible positive review in validation or test becomes a retrieval request. The ranker excludes the user's earlier reviewed items, searches the full train-known catalog, and counts a target absent from that catalog as a miss.
Lifetime and recent popularity compete on validation to select the control. The challenger blends that control with co-review scores; validation also selects its blend weight and decides whether a user-cluster bootstrap supports opening the route. This choice is saved before the test archive is read. The design avoids selecting a winning model from test outcomes, although it cannot remove the benchmark's retrospective filtering or turn reviews into exposure logs.
Held-out evidence
The selected method was tested on the previously unopened Musical Instruments category. Across 35,905 eligible requests, the hybrid reached 0.008681 NDCG@10 versus 0.008007 for recent popularity. The paired hybrid-minus-popularity difference was +0.000674, with a user-cluster 95% interval of [0.000040, 0.001338].
The Musical Instruments gain is positive but small, and both absolute scores are low. Of its 35,905 eligible test requests, 12,305 targeted items absent from the training catalog. Reordering known items cannot recover those targets. The experiment therefore exposes a coverage problem as well as a ranking result.
Neural challenger
An exploratory extension trains a two-tower retriever on 313,522 chronological training pairs. One tower encodes up to 20 prior positive items; the other embeds products. Training uses in-batch negatives while masking duplicate targets and known positives. The fitted model scores the full train-known catalog, then validation selects a blend with recent popularity and compares it with the already active co-review hybrid.
The standalone neural model scored 0.001411 NDCG@10 on validation. A 40% neural blend reached 0.011774, ahead of recent popularity at 0.010715 but behind the hybrid at 0.011969. Its paired interval against the hybrid crossed zero, so the gate kept the hybrid active. On the previously studied test period, the neural blend scored 0.008357 versus 0.008681 for the hybrid. This is a development comparison, not a second independent confirmation: the category's test result was already known before the neural model was designed. The interactive neural view above separates validation from that exploratory test result.
When earlier words fail to recommend
A later arm asked whether a person's earlier review prose could improve the frozen behavioral ranking. Its measured configuration could only re-rank items in the behavioral shortlist; parameters were selected on validation. On the full Musical Instruments test split, that own-words arm lost 0.001678 NDCG@10 with lexical retrieval (95% interval [−0.002318, −0.000980]) and 0.001282 with encoded retrieval ([−0.001944, −0.000577]). The lexical version also lost 0.002858 on Video Games ([−0.003598, −0.002195]). These extensions are exploratory because both category test periods had already been opened by earlier work. Neither route earned promotion.
Two diagnostics help explain the failure without reducing it to a single cause. The target was in the promotable shortlist for 33.21% of Musical Instruments requests, and unconstrained lexical search using earlier review prose retrieved it in 2.42% of requests with usable wording. Those rates have different denominators and cannot be multiplied into a measured joint ceiling. An exact target title retrieved its product in 99.98% of eligible cases, but that target-informed probe does not establish quality on natural queries.
An expanded external-pair diagnostic reached the target in 61.57% of 1,983 matched Musical Instruments requests with lexical retrieval, and in 67.93% with encoded retrieval. The lexical full-split difference was +0.004931: +0.002500 came from observed ESCI shopping queries and +0.002431 from Amazon-C4 rewrites of target-product reviews. The corresponding Video Games lexical difference was +0.006167; the Musical Instruments encoded difference was +0.004232. Each phrase was attached using the known target, and the ESCI source includes Exact and Substitute judgments from both splits. These results show that suitable words can move the frozen ranker when a target-linked phrase is supplied, but they do not establish a benefit from wording available at request time. The earlier smaller ESCI slice used different source rules, so its negative contribution cannot be explained by sample size alone. The aggregate results and source audit report the denominators, intervals, and all six wording conditions.
From measurement to a usable system
A local HTTP service serves the same ranking function through a hash-checked model bundle. It can also load the neural weights after source and integrity checks, returning that ranking as a shadow while the validation gate stays closed. For each request it reports the active route alongside baseline and shadow rankings; empty or unsupported histories and challenger failures return the baseline with a reason. A separate endpoint answers a typed phrase against product text without displacing the validated behavioral route. The interactive walkthrough above uses invented users, item names, and precomputed rankings to illustrate core routing; the phrase endpoint runs in the local service. The source repository includes the synthetic demo and aggregate results; downloaded reviews, product text, and fitted weights are not distributed.
Limits and next test
This is a category transfer within one platform and one fixed time split, not evidence of online impact. The 5-core subset uses support measured over the full corpus, and a review is neither an impression nor an unbiased engagement label. The paired interval resamples users in that split; it does not capture refitting, shared-item dependence, or future traffic shifts. One compact neural retriever is a useful comparison, but it does not exhaust modern retrieval methods. A useful next test would use exposure-logged or less-filtered activity, independent time periods, and a predeclared comparison against stronger baselines.