Bias & Debiasing

By Ludovico Bessi, Senior ML Engineer at Google · Updated

Logged recommendation data is biased: users click top positions more (position bias), only see what the system showed (exposure bias) and engage more with popular items (popularity bias). Debiasing corrects for this with propensity weighting, position-aware training, randomized exploration data and correction terms such as logQ.

Feedback Is Biased by the System Itself

Recommenders learn from what users did with what they were shown. That data is shaped by the system's own choices:

  • Users click top positions more, regardless of quality (position bias).
  • Users can only engage with items that were shown (exposure bias).
  • Popular items are shown more, get more engagement, and are shown even more (popularity bias).
  • Training sets contain only items that survived earlier funnel stages (selection bias).
  • Users who rate or engage explicitly are not representative (conformity and self-selection).

A model trained naively learns these biases as if they were preferences, and the feedback loop amplifies them.

Position Bias

The problem

An item at position 1 might get several times the clicks it would get at position 5. A model trained on raw clicks learns that "items the old ranker put at the top are good", partly circular.

Estimating it

  • Randomization: randomly swap results in a small slice of traffic (for example, swap position k with position 1) and measure click-through by position. Accurate but costs some user experience.
  • Intervention harvesting: use natural variation in positions across different rankers or experiments to estimate position effects without explicit randomization.

Correcting it

  • Inverse propensity weighting (IPW): weight each click by 1 / P(examined at that position). Clicks at low positions, which were less likely to be seen, count more. This is the basis of unbiased learning to rank (Joachims et al., 2017).
  • Position as a feature, removed at serving: train with position as an input so the model attributes position effects to that feature, then serve with position set to a fixed value (or omitted).
  • Shallow position tower: YouTube's "Recommending What Video to Watch Next" (2019) added a separate shallow tower taking position and device features, whose output is added to the main model's logit during training and dropped at serving. The main model learns relevance; the shallow tower absorbs position bias. Dropout on the position feature prevents the main model from over-relying on it.

Exposure and Selection Bias

Items never shown have no feedback, positive or negative. A model trained only on shown items:

  • Has never seen the kinds of candidates retrieval produces but ranking rejects.
  • Treats "not shown" as missing, but often implicitly as negative.

Mitigations:

  • Exploration: show a small fraction of traffic items chosen with some randomness, logging propensities, so every item type gets some unbiased data.
  • Counterfactual learning: train with IPS-weighted objectives using logged propensities (see Offline vs Online Evaluation).
  • Entire-space modeling: as in ESMM, model dependent outcomes over all impressions, not just clicked ones (see Multi-Objective Optimization).
  • Ranking-stage negatives in retrieval: train retrieval with candidates that were retrieved but not shown as additional negatives (carefully, since some may be false negatives).

Popularity Bias

Popular items dominate training data, so models over-recommend them; long-tail and new items under-perform.

Corrections:

  • logQ correction in retrieval: subtract the log sampling probability of each item when using in-batch negatives (see Two-Tower Models).
  • Popularity-aware negative sampling: sample negatives in proportion to popularity raised to a power below 1.
  • Debiased training objectives: causal approaches that separate the popularity effect from user–item match during training, then remove or control it at inference.
  • Re-ranking for coverage: guarantee exposure for new or niche items (see Re-ranking & Business Logic).

Not all popularity is bias: popular items are often popular because they are good. The goal is to stop the model from learning more popularity than the data justifies.

Duration and Content-Length Bias

Watch time favors long videos (more seconds possible); completion rate favors short ones. Normalizing engagement by duration buckets — for example, predicting the percentile of watch time within videos of similar length — avoids systematically favoring one length.

Debiasing Evaluation, Too

If evaluation data has the same biases, a biased model looks great offline. Use:

  • Randomized traffic slices as unbiased evaluation sets.
  • IPS-weighted offline metrics.
  • A/B tests as the final arbiter.

Practical Checklist

  1. Log positions, surfaces and, where possible, propensities for every impression.
  2. Keep a small randomized or exploration traffic slice; it powers both debiased training and evaluation.
  3. Model position explicitly (shallow tower or feature removed at serving).
  4. Apply logQ correction and popularity-aware sampling in retrieval.
  5. Normalize duration-dependent labels.
  6. Monitor coverage and new-item exposure to see bias in production.

Key Takeaways

  1. Logged feedback reflects what was shown and where, not just what users prefer.
  2. Position bias is corrected with propensity weighting or a separate position tower dropped at serving.
  3. Exposure and selection bias need exploration traffic, logged propensities and entire-space modeling.
  4. Popularity bias is reduced with logQ correction, sampling choices and re-ranking for coverage.
  5. Evaluate on debiased data too, or biased models will look best offline.