← AI HoodmanNo. 012 · Retrieval

Recommendation Funnel

How do you pick 10 videos from a billion in 100 ms?

01of 07
Use the arrows, swipe or the arrow keys to turn the slides. Click a slide to read it page by page.
  1. Slide 1 of 7: You have 1 billion videos and 100 ms to fill a feed. Your ranking model needs 0.01 ms per video. 2.8 hours 0.01 ms per video 100 ms Needed Budget 1 billion videos How do you pick which videos to rank?
  2. Slide 2 of 7: The key idea: Use a funnel. 1 billion videos 1,000 All videos The whole catalog Retrieval Fast and rough Picks 1,000 Ranking Top 10 Slow and precise Scores all 1,000, keeps top 10 Retrieval shrinks the pool cheaply, so the expensive model only scores a few videos.
  3. Slide 3 of 7: Retrieval: A two-tower model. Match score Dot product User embedding Video embedding User tower Video tower Watch history Searches Title Topic Location Device Creator Length User features Video features The towers never see each other, so every video's embedding can be computed ahead of time.
  4. Slide 4 of 7: Training: Pull watched videos closer. Pulled closer Pushed away Watched by this user Watched by other users in the batch User The model learns to score the watched video above everything else.
  5. Slide 5 of 7: The catch: 1 billion comparisons is still too slow. User Group center Searched Skipped 1 billion videos → 100,000 groups of 10,000. Compare the user to every center, then search the closest 100 (0.1%): 100,000 + 100 × 10,000 = 1.1M comparisons. Raise the 0.1% until you find ~95% of the true best matches. This is approximate nearest neighbor search (FAISS, ScaNN). It may miss a few good videos, but 1 billion / 1.1M ≈ 900x less work.
  6. Slide 6 of 7: The whole request: Under 100 ms. About 25 ms 100 ms budget User tower 5 ms Retrieval: approximate nearest neighbors 10 ms Ranking: 1,000 videos → top 10 10 ms Each stage does only the work the next one needs.
  7. Slide 7 of 7: Takeaway: Retrieve fast and rough. Rank slow and precise.

Keep reading

Recommendation Funnel · 1 / 7
  1. Slide 1 of 7: You have 1 billion videos and 100 ms to fill a feed. Your ranking model needs 0.01 ms per video. 2.8 hours 0.01 ms per video 100 ms Needed Budget 1 billion videos How do you pick which videos to rank?
  2. Slide 2 of 7: The key idea: Use a funnel. 1 billion videos 1,000 All videos The whole catalog Retrieval Fast and rough Picks 1,000 Ranking Top 10 Slow and precise Scores all 1,000, keeps top 10 Retrieval shrinks the pool cheaply, so the expensive model only scores a few videos.
  3. Slide 3 of 7: Retrieval: A two-tower model. Match score Dot product User embedding Video embedding User tower Video tower Watch history Searches Title Topic Location Device Creator Length User features Video features The towers never see each other, so every video's embedding can be computed ahead of time.
  4. Slide 4 of 7: Training: Pull watched videos closer. Pulled closer Pushed away Watched by this user Watched by other users in the batch User The model learns to score the watched video above everything else.
  5. Slide 5 of 7: The catch: 1 billion comparisons is still too slow. User Group center Searched Skipped 1 billion videos → 100,000 groups of 10,000. Compare the user to every center, then search the closest 100 (0.1%): 100,000 + 100 × 10,000 = 1.1M comparisons. Raise the 0.1% until you find ~95% of the true best matches. This is approximate nearest neighbor search (FAISS, ScaNN). It may miss a few good videos, but 1 billion / 1.1M ≈ 900x less work.
  6. Slide 6 of 7: The whole request: Under 100 ms. About 25 ms 100 ms budget User tower 5 ms Retrieval: approximate nearest neighbors 10 ms Ranking: 1,000 videos → top 10 10 ms Each stage does only the work the next one needs.
  7. Slide 7 of 7: Takeaway: Retrieve fast and rough. Rank slow and precise.