When Sequence Models Earn Their Complexity: A Basket-Level Evaluation of Rating Activity
Abstract
MovieLens rating timestamps record when users rated items, not what they saw. This study tests whether a small causal Transformer improves full-catalog recovery of the next positive-rating basket over a GRU under a shared train-only catalog, validation selection, and popularity fallback. Across 961 eligible held-out users, the seed-averaged Transformer-minus-GRU difference in macro-user NDCG@10 was +0.014002, with a paired user-cluster 95% interval of [+0.008097, +0.020201]. The Transformer reached 0.050921 NDCG@10, only 0.001315 above validation-selected recent popularity. In single-seed diagnostics, permuting complete timestamp baskets did not reduce either sequence model's held-out score. The result supports a narrow offline comparison, not a claim about viewing behavior, online lift, or a new recommendation algorithm.
The Transformer beat the GRU on the study's declared outcome. That result has a useful qualification: its average score was 0.050921 NDCG@10, while validation-selected recent popularity reached 0.049606. The difference between the two is only 0.001315. The paper asks what the sequence model earns once a simple control, model-selection boundaries, and local forward cost are visible together.
Explore the comparison
Where the sequence helps, and where it does not
Compare fixed held-out measurements, inspect a single-seed order diagnostic, then trace an invented rating history through the input rule. Controls never change the measured results.
Held-out MovieLens 1M test
The baseline nearly closes the gap
Validation selected the model settings. Three trained seeds are averaged for each neural family. All methods rank the same train-known catalog for 961 eligible users.
NDCG@10. Bars start at zero and share a 0 to 0.055 scale.
Transformer minus GRU
+0.014002
Paired user 95% interval: [0.008097, 0.020201]
Transformer minus recent popularity
+0.001315
Descriptive gap; no predeclared interval for this contrast.
Local batch-one forward time
Median process-level p50 across five fresh processes. Encoder and full item logits only.
GRU
2.799 ms
Transformer
1.736 ms
These offline rating-recovery scores are not exposure-adjusted relevance or online lift. Forward time excludes candidate filtering, sorting, transport, and service overhead.
Seed-11 diagnostic
Did the original order help?
The order intervention retrained a model after permuting complete timestamp baskets. The target and training prefix count stayed fixed. Choose a family to compare its held-out scores.
Held-out NDCG@10, zero-based bars on a common 0 to 0.055 scale.
What this diagnostic permits
Permuting baskets did not lower the score in either seed-11 run. That weakens a claim that the original chronological order explains the Transformer result. Retraining also changes optimization, and the no-position Transformer still has a causal mask, so this is not a causal proof that order is irrelevant.
Invented titles and histories
What can the encoder see?
Select a synthetic rating history. The future positive basket is never input to the model. This walkthrough contains no real user record and does not calculate recommendation quality.
At the forecast origin
Two usable positive-rating timestamps remain before the forecast origin.
- t1Atlas Drift + Comet Circuit
- t2Signal Runner
Route: sequence model
Causal attention access
A row may read its own basket and earlier baskets. It cannot read a later one.
| Read | t1 | t2 |
|---|---|---|
| t1 | ✓ | × |
| t2 | ✓ | ✓ |
Items sharing a timestamp make one unordered basket. An unknown input item is removed from the history, while an unknown future target remains a miss in the offline metric.
A rating history is not an exposure log
MovieLens 1M provides explicit ratings and their timestamps. A timestamp says when someone rated an item; it does not say when the item was shown or watched. The experiment predicts the first later timestamp basket containing a previously unrated item scored at least four. Ratings tied at one timestamp form an unordered basket. A user's earlier positive baskets supply the model input, and the future target basket cannot enter it.
All methods rank the same catalog of items known by the end of training. They exclude items the user rated before the forecast origin. Three held-out target items were absent from that training catalog; they remain metric misses because none of these rankers could retrieve them. When no train-known positive basket remains in a user's history, the sequence models take the same recent-popularity fallback.
A comparison frozen before test
The locally retained protocol specifies an 80/10/10 chronological split, a mean encoder, a GRU, a causal Transformer, popularity, and item-neighborhood controls. It also specifies the validation grid, three selected-model seeds, order interventions, a paired user bootstrap, and a 12-hour GPU-training cap. All 27 planned validation cells completed. Model settings and source hashes were sealed before the final test was opened once. The protocol files are not part of the public source repository.
Across 961 eligible users, the Transformer minus GRU difference was +0.014002 macro-user NDCG@10. The paired 95% interval was [+0.008097, +0.020201]. This interval compares those fitted architectures on this split. It does not cover retraining variability, future traffic, or the small descriptive gap between Transformer and popularity. The paper and aggregate technical report show the full results and failure accounting.
What the order check changes
The seed-11 GRU scored 0.039760 with original basket order and 0.039823 after whole-basket permutation. The corresponding Transformer scores were 0.046206 and 0.052260. A no-position Transformer reached 0.050369, although it still retained a causal mask. These retrained interventions are diagnostic. They weaken the argument that the original chronological order caused the Transformer's result, but they do not prove order is irrelevant.
Five fresh local processes measured the selected models' batch-one encoder and full-item-logit forward path. Median CPU p50 was 1.736 ms for the Transformer and 2.799 ms for the GRU; CUDA p50 was 3.679 ms and 4.968 ms. These timings omit candidate filtering, sorting, and service overhead. The Transformer had more parameters yet a faster measured forward path on this hardware, so parameter count alone would have been a poor cost proxy.
The study is an offline recovery test on one rating dataset and one chronological split. It does not estimate what people saw, what they would click, or whether either model would improve a live system. The source repository includes the separate paper, synthetic walkthrough, tests, aggregate outputs, and verification record. Raw rating rows, user-level predictions, and fitted weights stay local under the dataset owner's terms. This MovieLens study has a separate data and evidence boundary from the Amazon Reviews Recommendation Decision Lab.