Offline vs Online Evaluation

By Ludovico Bessi, Senior ML Engineer at Google ยท Updated

Offline evaluation replays logged data to score a model cheaply; online evaluation runs an A/B test on real users and is the only ground truth. Offline metrics often mislead because logs are biased by the old system, so the craft is making offline results predictive and A/B tests statistically trustworthy.

Two Ways to Evaluate

  • Offline evaluation replays logged data: train on the past, predict a held-out period, compute metrics. Fast, cheap, safe, and repeatable.
  • Online evaluation shows the new system to real users in a controlled experiment (A/B test) and measures what they do. Slow, expensive, risky โ€” and the only ground truth.

Good teams use offline evaluation to decide what is worth testing online, and online tests to decide what ships.

Offline Evaluation Done Right

Split by time, not at random

Recommendation data has strong temporal structure. Randomly splitting interactions leaks the future into training: the model sees tomorrow's trending video while "predicting" today. Always train on a period and evaluate on the period that follows, mirroring production.

Evaluate what production sees

Offline datasets often contain only impressions the old system chose to show. A retrieval model evaluated only on those items cannot get credit for finding good items the old system missed โ€” and cannot be penalized for retrieving bad ones that were never shown. Mitigations:

  • Evaluate retrieval on all engaged items, including those reached through search or external links, not just recommended ones.
  • Use sampled metrics carefully. Computing recall@K against 100 random negatives instead of the full catalog is cheap but can change which model looks better; research has shown sampled metrics can be inconsistent with full-catalog metrics. Prefer full-catalog evaluation for final decisions.

Counterfactual (off-policy) evaluation

The fundamental problem: logged data tells you what users did with what the old policy showed. You want to know what they would have done with what the new policy would show.

Inverse propensity scoring (IPS) reweights logged outcomes by how much more (or less) likely the new policy is to show each item than the old policy was:

Vฬ‚(new) = mean over logs of [ reward ร— ฯ€_new(item | context) / ฯ€_old(item | context) ]

This is unbiased if the old policy's propensities are known and every item the new policy might choose had some chance of being shown. In practice the weights can be huge (high variance), so teams clip them, use self-normalized IPS, or doubly robust estimators that combine a reward model with IPS corrections. Logging propensities โ€” and adding a little randomization (exploration) to the production policy โ€” is what makes this possible.

Replay and simulation

  • Replay: run the new policy over logged requests and only count events where it would have chosen the logged item. Unbiased under uniformly random logging, but discards most data otherwise.
  • Simulators: model user behavior and simulate sessions. Useful for testing long-term or slate effects, but only as good as the user model.

Online Evaluation: A/B Testing

The basics

Randomly assign users (not requests โ€” users, so each person has a consistent experience) to control or treatment. Run long enough to detect the effect size you care about. Compare metrics with proper statistical tests.

Sizing

The required sample size grows with metric variance and shrinks with the square of the effect size. Many recommendation improvements are fractions of a percent, so tests need very large populations or variance reduction:

  • CUPED (Controlled-experiment Using Pre-Experiment Data) uses each user's pre-experiment behavior as a covariate to remove predictable variance, often cutting required sample size substantially.
  • Choose less noisy metrics for decisions (for example, capped watch time rather than raw).

Pitfalls

  • Peeking: stopping a test as soon as p < 0.05 inflates false positives. Fix the duration up front or use sequential testing methods.
  • Multiple comparisons: tracking 50 metrics means a few will "move" by chance. Pre-register primary metrics.
  • Novelty and primacy effects: look at how the effect evolves over time, not just the average.
  • Network and marketplace interference: in two-sided systems, treatment can affect control (a creator boosted in treatment gets fewer impressions in control). Cluster-randomized or switchback designs help.
  • Feedback-loop leakage: if treatment and control models train on pooled logs, a better treatment can improve control through shared data, shrinking the measured difference.

Interleaving

For ranking changes, interleaving merges results from two rankers into one list per user and credits whichever ranker's items get engaged. Because each user compares both systems directly, interleaving can detect preference differences with far less traffic than A/B tests. Netflix and others use it as a fast first filter before full A/B tests.

A Practical Evaluation Ladder

  1. Offline metrics on a time-based split, full-catalog where possible.
  2. Counterfactual estimates using logged propensities, if available.
  3. Shadow traffic: run the new model on live requests without serving results, to check latency, coverage and prediction distributions.
  4. Interleaving for ranking changes, to filter candidates quickly.
  5. A/B test with pre-registered metrics, CUPED, and a fixed duration.
  6. Long-term holdout for launches that may have slow effects.

Key Takeaways

  1. Offline evaluation is fast but biased by the logging policy; online A/B tests are the ground truth.
  2. Split data by time and evaluate retrieval on all engagements, not only recommended ones.
  3. IPS and doubly robust estimators correct for the logging policy, if you log propensities and explore a little.
  4. Use CUPED, fixed durations and pre-registered metrics to make A/B tests trustworthy.
  5. Interleaving detects ranking preferences with much less traffic.