Machine learning at scale 🤖
← Back to book index

Ad Selection and Serving

The final steps: ensuring selected ads are valid and serving them reliably.

Final Checks: Frequency Caps, Budget Exhaustion, Real-Time Signals

Frequency Caps

Limit how often a user sees the same ad:

  • User-level caps: Per-user impression limits
  • Campaign-level caps: Overall campaign frequency
  • Time windows: Daily, weekly, or lifetime caps

Implementation:

  • Check against user's recent ad history
  • Approximate counting for scale (Bloom filters, count-min sketches)
  • Real-time updates as ads are served

Budget Exhaustion

Verify advertiser has remaining budget:

  • Campaign budget: Total budget for campaign
  • Daily budget: Daily spending limits
  • Real-time tracking: Update spend as ads are served

Challenges:

  • Race conditions: Multiple servers serving simultaneously
  • Overspend prevention: Stop serving before budget exhausted
  • Smooth pacing: Spend budget evenly over time (covered in Chapter 15)

Real-Time Signals

Last-minute checks that can't be done earlier:

  • User status: Has user become ineligible? (e.g., age, location change)
  • Ad status: Has ad been paused or disapproved?
  • Inventory availability: Is the ad slot still available?

The Serving Stack: Latency, Reliability, and Fault Tolerance

Latency Requirements

  • Total budget: <100ms from request to served ad
  • Network: 20-40ms for request/response
  • Processing: 50-70ms for all checks and serving
  • CDN: Edge caching for ad creatives

Reliability

  • 99.9%+ uptime: Ads must serve even during partial failures
  • Graceful degradation: Fallback to simpler serving when components fail
  • Redundancy: Multiple servers, regions, data centers

Fault Tolerance

  • Circuit breakers: Stop calling failing services
  • Timeouts: Don't wait indefinitely for slow services
  • Default ads: Serve fallback ads when primary system fails
  • Monitoring: Alert on errors and latency spikes

Logging for ML: What to Capture and Why

Essential Logs

Request Context

  • User ID, session ID, request timestamp
  • Page context, device, location
  • User features used in models

Candidate Information

  • Which ads were retrieved
  • Filtering decisions and reasons
  • Scores and rankings

Prediction Logs

  • Model predictions (CTR, CVR, etc.)
  • Feature values used
  • Model version and calibration

Auction Outcome

  • Winning ad and price
  • All bids and scores
  • Reserve price and floor

Serving Result

  • Ad actually served (may differ from winner due to failures)
  • Creative ID and format
  • Serving timestamp

User Feedback

  • Impressions (ad was shown)
  • Clicks (user interacted)
  • Conversions (desired actions)
  • Negative feedback (hide, report)

Why This Matters

  • Model training: Need labels (clicks, conversions) with features
  • Debugging: Trace why specific ads were shown
  • Attribution: Connect revenue to specific auctions
  • Monitoring: Detect issues and anomalies
  • Experimentation: Analyze A/B test results

Logging Best Practices

  • Completeness: Log everything needed for analysis
  • Efficiency: Don't log unnecessary data (costs money)
  • Privacy: Respect user privacy, anonymize where needed
  • Retention: Keep logs long enough for model training but not forever
  • Schema evolution: Handle changes in log format over time

Proper logging is the foundation for all ML improvements and system debugging.

Content to be expanded...