Sequential Generative Recommenders (HSTU)

By Ludovico Bessi, Senior ML Engineer at Google Ā· Updated

Sequential generative recommenders treat a user's history as a sequence of tokens and train a large transformer to predict the next action, replacing hand-engineered features. Meta's HSTU showed this approach scales with compute like language models, and semantic IDs (TIGER) let models generate item identifiers directly.

From Features to Sequences

Classic deep rankers (DLRM-style) depend on hundreds of hand-engineered features: counters, crosses, aggregated histories. They work, but they have two limits:

  1. Feature engineering is a bottleneck — each new signal needs pipelines, backfills and tuning.
  2. They do not scale well with compute. Making DLRM-style models bigger tends to plateau, unlike language models, which keep improving as they grow.

Generative recommenders reframe the problem the way language models do: a user's history is a sequence of tokens (items and actions), and the model learns to predict what comes next. Features become implicit in the sequence, and quality can scale with model size.

HSTU: Hierarchical Sequential Transduction Units

Meta's 2024 paper "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations" (Zhai et al.) introduced Generative Recommenders (GRs) built on a new architecture, HSTU.

The formulation

  • Each user's interactions form one long chronological sequence of items interleaved with actions (watched, liked, skipped…).
  • Retrieval becomes predicting the next item in the sequence.
  • Ranking becomes predicting the user's action on a candidate item placed at the end of the sequence (target-aware, like DIN, but inside a single sequence model).
  • Most heterogeneous features (counters, categories) are dropped or folded into the sequence; the model learns them from raw history.

Architecture changes versus a standard transformer

Recommendation data differs from text: vocabularies are huge and constantly changing, and the intensity of a user's preference matters, not just its direction. HSTU adapts the transformer block:

  • Pointwise aggregated attention: replaces softmax normalization with a pointwise non-linearity (SiLU), so attention weights are not forced to sum to one. This preserves how strongly a user engages with related items, which softmax normalization would wash out.
  • Relative attention bias using both position and time gaps between actions.
  • Gating: an elementwise gate (similar to gated linear units) folds the feed-forward network into the attention block, making the layer simpler and cheaper.

Making it trainable and servable

  • Stochastic length: randomly sub-sampling long user sequences during training cuts attention cost substantially with little quality loss, exploiting the sparsity of user histories.
  • Efficient kernels for long sequences.
  • M-FALCON: at inference, many ranking candidates are scored in one pass by appending them to the user sequence with attention masks that keep candidates independent, amortizing the cost of encoding the history across candidates.

Results

The paper reported that HSTU-based GRs outperformed strong baselines substantially on public datasets, that GRs with up to 1.5 trillion parameters improved online metrics in A/B tests on Meta's platforms, and — most importantly — that model quality followed a power-law scaling with training compute across a wide range, similar to language models. That finding shifted the industry toward scaling recommendation models.

Semantic IDs and Generative Retrieval

Item IDs are arbitrary integers: item 51,234 tells the model nothing about item 51,235. Semantic IDs fix that.

  • Encode each item's content into an embedding, then quantize it into a short tuple of discrete codes (for example with a residual-quantized VAE). Similar items share code prefixes.
  • A sequence model generates the next item's semantic ID token by token, using beam search to produce several candidates.

Google's TIGER (2023) introduced this generative retrieval approach for recommendation. Benefits:

  • No ANN index; retrieval is decoding.
  • Knowledge sharing across similar items via shared code prefixes, which helps cold start.
  • Bounded vocabulary: a few thousand codes instead of billions of IDs.

Challenges include ID collisions, invalid generated codes, and keeping semantic IDs stable when the catalog or encoder changes.

End-to-End Generative Recommenders

Kuaishou's OneRec goes further: a single encoder–decoder model generates a session of recommended items directly, replacing the retrieval-and-ranking cascade, and is fine-tuned with preference alignment (a DPO-style method driven by a reward model) to optimize what users actually value. It shows that the classic funnel can, at least in some products, be collapsed into one generative model. For a breakdown, see the newsletter post on OneRec.

Trade-offs

DLRM-style ranker Generative recommender
Inputs Many engineered features Raw interaction sequence
Scaling with compute Plateaus Power-law improvements reported
Serving cost Per-candidate MLP Long-sequence attention; needs amortization tricks
Freshness of item vocabulary Easy (new ID rows) Harder (IDs or semantic IDs must be learned)
Maturity Very mature tooling Rapidly evolving

Generative recommenders are not a free upgrade. Long sequences are expensive to serve, real-time features such as "trending in the last ten minutes" are harder to express, and the engineering stack is less mature. Many teams adopt them incrementally: generative retrieval as one more retrieval path, or sequence models feeding embeddings into an existing ranker.

Key Takeaways

  1. Generative recommenders treat user histories as sequences and predict the next item or action, replacing much feature engineering.
  2. HSTU adapts transformers to recommendation with pointwise attention, time-aware relative bias and gating.
  3. Stochastic length and M-FALCON make training and serving long sequences affordable.
  4. HSTU showed recommendation quality can follow power-law scaling with compute.
  5. Semantic IDs enable generative retrieval without ANN indexes; OneRec shows end-to-end generation can replace the funnel.