Contents
  1. Purpose and development
    1. Useful items for a particular task
    2. The complete selection policy
    3. Turning points in recommendation
  2. Feedback and personalization
    1. What an interaction records
    2. Attributes and collaborative behavior
    3. Current intent and user control
  3. Candidates, scores and lists
    1. Candidate coverage
    2. What the ranking score means
    3. Composing the displayed list
  4. Learning from selected feedback
    1. How exposure shapes learning
    2. Starting without enough history
    3. Exploration under constraints
  5. Evidence for a changed policy
    1. Reconstructing an honest offline test
    2. Learning from another policy's logs
    3. Measuring product benefit
  6. Keeping recommendations useful
    1. Shared models and serving choices
    2. Update the right boundary
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Recommendation systems

Recommendation systems select useful options from a catalog for a person or situation. They combine representations of users and items with retrieval, ranking and list composition, then learn from the responses their choices make possible. Building one means choosing what usefulness means, handling sparse or selective feedback and testing whether recommendations improve the experience. This chapter connects modeling choices to exposure, exploration and product outcomes.

Purpose and development

Useful items for a particular task

Usefulness depends on what the person is trying to do now. A video service can select from the same catalog, its collection of available items, for both a homepage and a next-video panel. These product surfaces—the places where choices appear—serve different tasks. YouTube describes watch history as a major homepage input and the currently watched video as the main next-video signal. Recommendations often work without an explicit search query, so the surface and activity within a session, an episode of use, help specify what to offer.

Define success before selecting a model. User benefit is the improvement the person seeks; a business outcome is the improvement the organization seeks. A proxy is an observable measurement used to approximate either one. A constraint specifies what the system must not violate while pursuing them. These distinctions matter because choosing, clicking and consuming are intermediate actions: their value depends on what happens afterward.

Two applications make the difference between an immediate action and an intended outcome concrete.
ApplicationImmediate observationOutcome that mattersDecision boundary
Garment sizingA customer purchases a recommended size.The garment fits; avoidable returns decline.Item sizing and personal fit preferences both matter.
Algebra instructionA student opens a recommended video.The student improves on later assessments.The student can ignore the recommendation.

Zalando's sizing account combines purchase and return behavior with fit feedback and specialist assessments. In the Math Nation field experiment, personalized video selection was compared with a predetermined curriculum sequence. The initial analysis found no significant treatment coefficient; missing covariates excluded 5,448 of 16,406 assigned students, and later usage-subgroup analyses were exploratory. Personalization describes a selection method, not an established benefit.

The complete selection policy

Eligibility determines whether an item may appear. A candidate is an item shortlisted for further consideration. Ranking assigns relative priority to candidates; list composition chooses which ones appear together. A slate is that displayed collection, often ordered. A recommendation policy includes the complete selection and presentation rule, including constraints and any randomness—not merely the fitted ranking model.

Eligibility is Boolean; preference is comparative. An unavailable item does not become permissible because it receives a high score. Conversely, an eligible item can disappear because no generator retrieves it or because a candidate budget excludes it. Finally, individually attractive candidates may be redundant when displayed together. These are different failure locations and require different repairs.

Stages manage computation. When scoring every eligible item is affordable, exhaustive comparison is possible. As the catalog or scoring cost grows, candidate generation reserves expensive evaluation for a smaller set. Spotify's 2025 account describes this narrowing across playlists, search and other surfaces. Candidate generation in Search and Retrieval explains the general shortlist mechanism; here the distinctive problem is choosing whose interests and which current context should shape it.

Turning points in recommendation

Recommendation developed through complementary answers to information overload, sparse judgments and expensive selection. Early collaborative systems made other people's judgments useful without requiring every reader to describe their interests. Later work changed which relationships were computed, how they were represented, and which outcomes the system tried to predict.

DevelopmentContribution
GroupLens — October 1994Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom and John Riedl's MIT–Minnesota system predicted Usenet article ratings from agreement among readers. It extended topic subscription with shared judgments. Original paper.
Amazon item-to-item recommendation — January–February 2003Greg Linden, Brent Smith and Jeremy York described precomputing relationships from shared purchases. Request-time lookup combined items related to a customer's history or current cart, avoiding a fresh search across the customer population. Implementation report.
Netflix Prize — launched October 2006The competition standardized prediction of withheld star ratings. Its target was a 10% reduction in rating-prediction root mean squared error relative to Cinematch—not 10% more satisfaction. Organizers' account.
Latent-factor models — August 2009 accountYehuda Koren, Robert Bell and Chris Volinsky explained compact user–item compatibility factors, biases, implicit feedback and temporal effects in their Prize work. Their ensembles combined many predictors rather than establishing one universally best model. Authors' article.
YouTube deep recommendation — 2016YouTube's research team described separate candidate generation and ranking, learning from viewing context and impressions rather than relying only on star ratings. System paper.
Netflix shared preference model — March 2025Ko-Jen Hsiao, Yesu Feng and Sudarshan Lamkhede described reusable representations learned from histories and content beneath downstream applications. The motivation included duplicated work across separately trained recommenders. Engineering account.

The continuing distinction is between evidence about related people, evidence about related items, and evidence about a particular decision. Precomputed item relationships remain useful when lookup is the right serving boundary. Learned representations can share information more broadly, but do not decide what counts as a successful recommendation. Even GroupLens recognized that evaluating on articles selected by one recommender could disadvantage another that would have selected different articles. Selected feedback is an old problem, not a side effect unique to today's models.

Feedback and personalization

What an interaction records

Explicit feedback is a deliberate judgment, such as a rating or dislike. Implicit feedback is behavior interpreted as a signal, such as watching, skipping or purchasing. YouTube lists satisfaction surveys alongside viewing behavior because watching does not exhaust the meaning of satisfaction. Repeated actions and dwell time likewise require a declared interpretation: the logged event is an observation, not direct access to intent.

A request, returned response, rendered card and measured view are distinct events. Define an impression by the instrumentation that produces it. Rendering alone does not prove attention. Amazon Personalize's documentation illustrates the distinction by removing out-of-stock shoes from returned recommendations before recording the displayed subset. An item without display confirmation has unknown exposure; a displayed item without interaction is not equivalent to an explicit rejection.

A delayed outcome adds another boundary. No purchase yet may mean no eventual purchase or simply insufficient follow-up. An observation is censored when the outcome remains unresolved at collection time. Define the outcome window before assigning negatives; the window defines the target, not what will happen forever. For a kept-purchase target, observing a purchase does not close its return window.

For training, features are information available when choosing; labels are the outcomes the model is asked to predict. The features-and-targets distinction prevents a later purchase or return from leaking into the earlier decision. A practical logging contract should join request and item IDs, permitted context, policy version, returned and displayed positions, event timestamps, and subsequent outcomes. Preserve repeated events rather than silently treating them as independent people. This is an application contract to design, not a universal vendor schema.

The cost of misinterpreting feedback depends on the decision. In the Indeed example discussed by Eugene Yan, experts labeled user–job fit because explicit negative feedback was sparse. A filter with 63% rejection precision meant that 63% of the recommendations it classified as bad were actually bad—not that it correctly rejected 63% of all recommendations. The remaining rejected matches illustrate the cost of using an unreliable negative judgment as an exclusion rule.

Attributes and collaborative behavior

Content-based filtering compares item properties with known interests. If a person requests short history documentaries, duration, subject and format provide evidence before those particular videos accumulate interactions. The representation limits what the system can express: missing attributes cannot contribute to the match, and item descriptions do not reveal an unknown person's preferences.

For a simple feature-based score, let uju_j express interest in feature jj, and vijv_{ij} describe item ii on that feature: s(u,i)=u⊤vi=∑j=1dujvij.s(u,i)=u^\top v_i=\sum_{j=1}^{d}u_jv_{ij}. Here dd is the number of features. Matching coordinates contribute to the score; it is not a probability of enjoyment.

Collaborative filtering instead uses patterns across people's recorded interactions. A user-based neighborhood finds people with similar histories and uses their preferences to suggest items. An item-based neighborhood finds relationships among items, then combines those relationships with the target person's history. Consider a customer interested in gardening: a new guide can match that attribute, while a buyer who purchased the same planter can supply a watering-can candidate. Shared purchases do not require matching product descriptions.

An embedding is a numerical representation used by later computation. Matrix factorization learns compact user and item embeddings whose comparisons approximate feedback, rather than assigning every coordinate an authored category. Missing interactions remain ambiguous, so the treatment of observed and unobserved entries affects what the learned compatibility means. User–item compatibility develops this distinction; vector comparison explains why dot product and cosine can produce different orders.

A hybrid recommender combines these evidence sources. Content can supply a plausible match where behavior is sparse; behavior can distinguish items with similar descriptions. One implementation represents customers, products and purchases as a graph, then follows purchase relationships to enrich text-based retrieval. The graph-based recommendation workshop demonstrates this approach; graph traversal explains how following recorded connections produces candidates. A co-purchase count remains a ranking signal, not a preference probability. Compare personalized methods with popularity-based selection to determine what they add beyond a shared default.

Two routes to a candidate

Attribute route

Target customer → declares gardening interest

Gardening attribute → B · new gardening guide

B has no recorded purchases. Its attribute match supplies a candidate.

Recorded purchase relationships

Target customer → A · planter

Another buyer → A · planter

Another buyer → C · watering can

Each solid arrow means “recorded purchase.” A is already-purchased context.

Candidate discovery: target → A → another buyer → C
This traversal follows the buyer-to-A purchase edge backward; it records no new purchase.

B · attributesC · shared purchases

No absent edge means dislike. The two candidates need neither matching descriptions nor an assigned fusion score.

Hypothetical customer and products. Attribute matching and shared purchases supply different information, not preference probabilities.

Current intent and user control

A durable preference and a current task are different inputs. Someone's established interest in documentaries does not determine what they want during every visit. Temporal models can separate stable patterns, gradual changes and transient effects. Recency weighting reduces an older interaction's contribution to a current prediction; it does not necessarily discard that interaction from training. Koren's temporal recommendation study found that older observations could retain useful collaborative structure even when predictions needed time-sensitive components.

Session state summarizes the current ordered activity. A session model can operate without a persistent user ID, updating that summary after each click and resetting it at a declared session boundary. Order carries information that a set loses: browsing cameras and then lenses presents a different latest context from browsing lenses and then cameras. This does not prove the person's intent; it makes the difference available to the model.

Explicit control can correct an inferred preference. In Personalization in the Era of LLMs, Shivam Verma describes natural-language playlist requests and edits to an exposed taste profile that influence recommendations. A request steers the current selection; a profile correction expresses a broader preference. Neither mechanism, as described, establishes that the edit changes an embedding or model weights. Context selection and parameter fitting are separate operations.

More history is not automatically better context. The 360Brew team reported improvement followed by a decline at its longest tested histories; the proposed explanation was limited long-context generalization, not a demonstrated loss of information. Choose the history budget by measured usefulness. Also establish permitted data use: avoiding permanent identity is possible, but removing direct identifiers does not necessarily anonymize released behavior. Narayanan and Shmatikov demonstrated linkage risks using sparse movie ratings and approximate dates.

Candidates, scores and lists

Candidate coverage

Candidate construction gives items an opportunity to compete. Content matching, item associations, recent behavior and popularity can supply different candidates. Combining sources is useful when their omissions differ, but the actual shortlist depends on source budgets, overlap and truncation. Candidate depth is the number allowed through to later scoring, not the number ultimately displayed.

Deduplicate overlapping sources by item identity, then distinguish mandatory exclusions from budget limits. In this example, two sources contribute B; D is unavailable; an explicit shortlist budget keeps A and B, leaving eligible C unscored. No later scoring improvement can put C into that request’s final list. Record source membership and exclusion reasons so these losses remain distinguishable.

Five entries, two candidates to score

Content source
ABC
3 entries
Behavior source
BD
2 entries
↓ Distinct union
ABcontent + behaviorCD
Deduplicate by identity
↓ Available
ABcontent + behaviorC
D removed: unavailable
↓ Ranker input
ABcontent + behavior
C removed: eligible; outside budget

Arrows mean set transformation. The example budget keeps A and B; it assigns no preference scores or universal filter order.

B is one item from two sources. D is unavailable; C is eligible but never scored under the example budget.

A two-tower model independently encodes the request and each item into compatible vectors. Item vectors can be indexed ahead of requests; the current request supplies the other vector. Approximate search may miss exact top candidates, so compare it with exhaustive scoring under the same setup. Dense retrieval explains the index mechanics.

Behavior should not silently become eligibility. The GraphRAG workshop warns that restricting retrieval to a sparse purchase neighborhood can exclude good content matches. An optional behavioral match preserves candidates lacking local purchase evidence. That is different from relaxing mandatory availability or authorization checks. Likewise, multiplying a similarity score by a purchase count is not self-justifying: their scales and zero values change the resulting order. Compare combination rules against labeled outcomes.

What the ranking score means

A ranker combines user, item and request information to estimate an outcome or relative preference. Its loss penalizes disagreement with the training target; changing that target changes the pressure applied during fitting. The general objective mechanism is shared across machine learning, but recommendation introduces a specific choice between predicting individual responses and learning an ordering.

Training familyTraining unitWhat receives pressure
PointwiseOne labeled user–item or query–item examplePredict its individual target.
PairwiseTwo items with a preferred order in the same contextRaise the preferred item's score relative to the other.
ListwiseA candidate list and its judgmentsImprove a list-level ordering objective.

These families need not imply different serving architectures. Pairwise training does not automatically emphasize the first position, and listwise training does not automatically encode diversity. Bayesian Personalized Ranking, introduced in 2009 by Steffen Rendle, Christoph Freudenthaler, Zeno Gantner and Lars Schmidt-Thieme, makes personalized ordering explicit. A training triple contains a user, an interacted-with item and an unobserved alternative; fitting favors the observed item. The alternative is assumed less preferred, not recorded as disliked. BPR's reported improvements used retrospective purchase and interaction evaluations, not an online demonstration of user benefit.

The weight assigned to each training example also changes what the score means. YouTube's 2016 ranking method weighted clicked impressions by watch time and unclicked impressions by one, giving longer watches more influence during fitting. For a model output p between zero and one, its odds are p/(1−p). Under the paper's approximation that click probability is small, the learned odds approximate expected watch time; the output is not an ordinary click probability. Calibration is a separate check: among comparable cases assigned a probability near p, the stated outcome should occur roughly a fraction p of the time. Check this before using predicted probabilities in expected-outcome calculations. See probabilities and decisions.

In this hypothetical fixed-context example, both garments are eligible. Different declared targets select different winners.
CandidatePredicted click probabilityPredicted purchase kept through the return window
A0.300.04
B0.200.07

Click ordering selects A; ordering by the specified kept-purchase outcome selects B. This establishes a decision difference, not measured improvement. The product must justify the target and validate its prediction. It can also combine several outcome estimates while enforcing separate constraints. Expensive joint scoring can remain a reranking operation on a shortlist; a more sophisticated scorer does not remove those policy choices.

Composing the displayed list

A list spends a limited number of positions. Three highly relevant items can still provide nearly the same choice. Intra-list diversity measures variation among selected items under a declared representation. Novelty concerns what is unfamiliar to the person, which incomplete history cannot establish perfectly. Serendipity adds experienced value: interview research describes refreshing or enriching discoveries, including rediscovery of forgotten interests. None is simply another name for high individual relevance.

Maximal Marginal Relevance builds a list incrementally. At each step, it balances a candidate's relevance against its similarity to items already selected. At the relevance-only endpoint it preserves relevance ordering; increasing the redundancy penalty favors different choices. An informal composition choice might replace one of several near-identical camera reviews with a relevant comparison of alternatives. Greedy MMR can change earlier positions too; it does not promise to preserve the first two choices. Whether that helps depends on whether the person wants breadth or depth.

A candidate’s score depends on the list so far

All five candidates remain eligible. At each step choose the greatest λ × relevance − (1 − λ) × maximum similarity to the selected set. For an empty list, similarity is 0. Equal scores use item ID order A–E.

Relevance only · λ = 1
A → B → C

Redundancy aware · λ = .55
A → C → E

Current slate: A → C → E · Before step 1: empty list · Choose A

Remaining candidateRelevanceMax similarityλ × relevancePenaltyScore
A · Camera X: full review ✓0.950.000.5220.0000.522
B · Camera X: quick review0.900.000.4950.0000.495
C · Landscape lens comparison0.800.000.4400.0000.440
D · On-camera audio guide0.720.000.3960.0000.396
E · Camera bag field test0.600.000.3300.0000.330
Fixed symmetric similarity matrix
ItemABCDE
A · Camera X: full review1.000.950.200.150.10
B · Camera X: quick review0.951.000.250.200.10
C · Landscape lens comparison0.200.251.000.350.10
D · On-camera audio guide0.150.200.351.000.20
E · Camera bag field test0.100.100.100.201.00
Fixed hypothetical relevance values and similarities; no learned similarity model or measured usefulness is claimed. Reduced redundancy need not improve the person’s experience.

The same problem appears above the item level. In Learning a Personalized Homepage, Chris Alvino and Justin Basilico's April 2015 account treats rows, titles and placement jointly. Independently ranking rows can repeat one interest across the page. Scoring each next row against those already selected reduces that redundancy, but further lookahead costs computation and need not find the optimal page. The account also separates freshness from stability: responding to new activity should not make recently suggested titles impossible to find again.

Selection and presentation require separate comparisons. Hold layout fixed when testing a replacement; hold item identities fixed when testing placement. Provider exposure is the attention opportunity allocated to an item's supplier, not the supplier's realized clicks or income. Exposure constraints can deliberately alter a relevance-maximizing order, especially when positions receive very different attention. Their rationale and tradeoffs must be explicit; a relevance model does not supply a fairness definition.

Learning from selected feedback

How exposure shapes learning

Exposure bias arises because previous selection determines which items can generate observed feedback. Position bias arises because placement changes the chance of examination. A click therefore reflects both the opportunity to notice an item and the response to it. High interaction counts cannot be read directly as stronger preference. This is a recommendation-specific case of selective labels.

The loop continues when observed responses become training data. A system may repeatedly favor well-observed items while leaving alternatives poorly understood. Its next model then learns from behavior partly caused by the previous policy, alongside the effects of preference and context. Prediction-induced feedback explains the broader principle. Recording every decision improves traceability but cannot reveal a person's response to an alternative never shown.

The policy helps create its training data

Observed response has both a selection path and a preference-and-context path.

Prior policyEarlier selection rules and fitted scores.
Selected item and positionThe opportunity created by the policy.
ExaminationWhether the person notices the item.
Preference and contextInterests and circumstances beyond placement.
Observed responseAn interaction or recorded noninteraction.
Later training dataResponses attached to selected opportunities.
Later policyA subsequent policy fitted using those records.
The feedback loop is unrolled across two policies. Prior selection affects examination and response; preference and context also affect response. Later fitting uses these selected observations, not outcomes for every possible item. These are selected dependencies, not a complete causal identification model.
Read the diagram as text
  • Prior policy. Earlier selection rules and fitted scores.
  • Selected item and position. The opportunity created by the policy.
  • Examination. Whether the person notices the item.
  • Preference and context. Interests and circumstances beyond placement.
  • Observed response. An interaction or recorded noninteraction.
  • Later training data. Responses attached to selected opportunities.
  • Later policy. A subsequent policy fitted using those records.
  • Prior policy → Selected item and position: chooses.
  • Selected item and position → Examination: affects attention.
  • Selected item and position → Observed response: determines offered item.
  • Examination → Observed response: enables interaction.
  • Preference and context → Observed response: affects response.
  • Observed response → Later training data: recorded as examples.
  • Later training data → Later policy: informs fitting.

Feedback effects are conditional, not a universal law of concentration. A simulation of repeated recommendation and retraining found increasingly similar consumption across users under its specified choice model. That similarity was distinct from concentration on globally popular items. In contrast, an August 2026 Netflix preprint reports a 60-day holdback experiment conducted in February–April 2025: twelve algorithmic updates, with no interface change, increased viewing and distinct titles played while reducing aggregate concentration. Attention shifted toward moderately popular titles, with little change for the least-popular half. The bundled intervention does not identify one responsible algorithm or establish lifetime welfare.

Feedback integrity is a separate concern. Shilling attacks fabricate coordinated rating profiles to promote or suppress items, using ordinary-looking filler ratings to influence collaborative relationships. This differs from independent noisy judgments because the records are strategically constructed. Treat behavioral inputs as an influence boundary, with the broader principles developed in AI Security.

Starting without enough history

Cold start means insufficient relevant interaction history. It is not one problem: a new user's interests may be unknown, a new item's appeal may be unknown, or a new service may lack population-level relationships. An opaque item identifier carries identity but no content information. Waiting does not solve the problem if the item never receives an opportunity to generate feedback.

Missing historyAvailable starting evidenceInitial policyWhat remains unknown
New userDeclared interests, current request, session actionsMatch the request; ask a small number of useful preference questions.Durable preferences beyond the supplied context.
New itemMetadata and contentInclude content-based candidates for relevant interests.How actual users will respond.
New serviceCatalog knowledge and editorial judgmentUse an executable curated or rule-based baseline.Collaborative relationships that have not yet been observed.

Preference elicitation asks for information that improves selection. An informative rating question is useless if the user does not know the item. Research on new-user onboarding therefore balances familiarity with disagreement among existing raters: familiar items are answerable, while divisive items can distinguish tastes. Adaptive selection uses previous answers to choose later questions. An unanswered prompt remains missing information, not a dislike.

Hybrid representations can change their dependence on evidence as it accumulates. Netflix's March 2025 design combines metadata-based and learned item-ID representations through an age-conditioned mixing layer. New titles rely more on metadata; established titles can use interaction-derived identity information. This lets behavior distinguish similarly described titles. It does not make age a universal confidence measure: old items can remain poorly observed, and metadata provides an initial prediction rather than proof of appeal.

Exploration under constraints

Exploitation chooses according to current outcome estimates. Exploration deliberately allocates opportunities to learn about uncertain alternatives. A contextual bandit observes request context, chooses an available action and receives a reward only for that action. Here an action can be a recommended item, and a reward is a declared measured outcome. It is not an intrinsic measure of user welfare. The LinUCB news-recommendation work makes the distinction concrete by combining estimated reward with an uncertainty bonus.

The allocation problem predates recommendation software. William R. Thompson's 1933 paper considered acting while treatment evidence was still accumulating, allocating according to the current probability that one option was better. The relevant contribution is continued learning under uncertainty—not a claim that modern recommenders existed then.

Epsilon-greedy usually exploits the highest estimate, reserving a fixed fraction for uniform random exploration. Thompson sampling instead maintains a posterior distribution—a distribution over plausible unknown outcome probabilities after observed evidence. It samples one plausible value per action and chooses the largest. A less-observed option can win a draw without having the highest estimated mean. This is uncertainty-aware allocation, not merely adding visually different items.

Illustrative pseudocode

Python-like pseudocode
# Basic binary-outcome Thompson sampling; eligible is nonempty.
# Each posterior represents an action's unknown success probability.
sampled = {a: posterior[a].sample() for a in eligible}
chosen = max(eligible, key=lambda a: sampled[a])

# Later, update only from the chosen action's resolved outcome.
if outcome_is_resolved(chosen):
    posterior[chosen].observe(binary_outcome(chosen))

For accountable learning, record context, available actions, the selected action, its selection probability and the eventual outcome. The sampled reward probability inside Thompson sampling is not the probability that the policy selects the action. Vowpal Wabbit's contextual-bandit tutorial exposes selection probabilities explicitly as part of the decision record.

Uncertainty can change the selected action

Fixed context, two eligible actions, binary outcomes. Each starts with a Beta(1, 1) prior. With s resolved successes and f failures, its posterior is Beta(1+s, 1+f). The mean is (1+s)/(2+s+f).

Posterior density over success probability, with a sampled value for each action. A is solid cyan, B dashed gold.Probability density4000.250.50.751Unknown success probabilityA drawB draw
Action / curveResolved s / fPosteriorMeanSampled value
A · solid cyan8 / 2Beta(9, 3)0.7500.715
B · dashed gold1 / 1Beta(2, 2)0.5000.750

Seed 3 · choose B, the largest eligible draw. Sampled values are success probabilities, not action-selection propensities.

Feedback applies only to the current chosen action, then advances the draw. Pending records stay unresolved and change neither posterior. Reset restores the reproducible seed-3 example: lower-mean B wins (.750 versus A’s .715). This sampler does not compute logging propensities.

Basic fixed-context Bernoulli example. The default lower-mean B win reflects posterior uncertainty, not superior measured usefulness. Eligibility, exposure limits and deployment stopping rules belong outside this sampler.

Deployment needs limits outside this sampling mechanism: enforce eligibility first, specify how much exposure uncertain options may receive, and define when adverse outcomes require stopping or review. Unresolved delayed rewards must not become immediate failures. These are application requirements, not a universal validated safety protocol. An adaptive bandit seeks useful allocations while learning; a randomized experiment can separately compare the complete bandit policy with another policy.

Evidence for a changed policy

Reconstructing an honest offline test

An offline test should reproduce the information boundary of the decision. With a temporal cutoff, later observations remain hidden; in event-by-event evaluation, predict each event before making it available to subsequent predictions. Use the catalog and eligibility that applied then. A point-in-time join attaches historical features from before each decision timestamp. Feast's implementation searches backward within a permitted age window. Such a join cannot repair a feature already computed using future information, and an event timestamp alone does not prove late-arriving data was available. See validation without leakage.

Compare against credible simple alternatives: popularity, recent-session rules or item neighbors, depending on the task. Then separate discovery from ordering. In the earlier candidate example, a test that judges only A and B cannot diagnose the omission of relevant C. Evaluate the generator against the declared eligible universe, and the ranker against its actual input; also measure the complete result.

These metrics separate finding relevant items from ordering them well. K is the rank cutoff: only the first K results count. State the evaluated population and how per-request measurements are aggregated.
MeasurementMeaningBoundary
Candidate recallFraction of relevant items admitted to later scoring.Judgments must include relevant items outside the shortlist.
Recall@KRelevant items in the top K divided by all relevant items in the evaluation universe.The denominator is not K. Specify treatment of lists with no relevant items.
Normalized discounted cumulative gain (NDCG@K)Assign a numerical gain to each relevance judgment, reduce its contribution at lower positions, and sum through rank K. Divide by the gain from ideally ordering the same judged candidates.Specify gain, position discount, ties and zero-ideal-gain handling; it does not measure diversity.
Catalog coverageDistinct recommended items divided by catalog size over a declared interval.More coverage need not mean more useful recommendations.

Keep the candidate universe and relevance judgments explicit when comparing recall or NDCG. Changing them changes the question, even if the metric's name stays the same. Metrics and denominators develops the calculations. Report new users, new items, infrequent users and low-exposure inventory separately when those are consequential deployment groups; a good aggregate can conceal the exact population the change was intended to help.

Ranking against sampled alternatives answers a different question from ranking against the catalog. Sampling changes how many stronger alternatives the relevant item encounters, and nonlinear ranking metrics can even reverse model comparisons in expectation under studied sampling schemes. Record the sampling rule and do not silently compare sampled scores with full-catalog scores. Temporal correctness also leaves exposure bias intact: an unobserved item still lacks a complete relevance judgment.

Learning from another policy's logs

Off-policy evaluation estimates a target policy's outcomes using feedback collected by a logging policy. A logging propensity is the probability that the logger selected an action in its context—not the predicted probability of a click and not the probability of examining a position. One approach, randomized replay, retains an event only when the target chooses the logged action. Its justification requires the stated randomized-logging and independent-event assumptions, and it discards unmatched observations.

Weighting instead adjusts each observed outcome for unequal selection opportunities. Let xx be context, aa the logged action, rr its observed reward, μ\mu the logging policy and π\pi the target policy. The contribution uses the ratio of their action-selection probabilities:

w=π(a∣x)μ(a∣x),weighted contribution=wr.w=\frac{\pi(a\mid x)}{\mu(a\mid x)},\qquad \text{weighted contribution}=wr.

For example, a logger chooses A with probability 0.8 and B with 0.2; a target chooses each with 0.5. Observed A rewards receive weight 0.625, and B rewards receive 2.5. The rarer opportunity contributes more. Reliable estimates require correct probabilities, comparable outcomes and support: every target action must have a nonzero logging chance. Rare actions create large, variable weights; zero support leaves outcomes unidentified. Clipping limits influence but changes the estimator and generally introduces bias. Causal foundations explain the assumptions behind such adjustment.

When the reward concerns the whole ordered slate, the action is the slate. Individual-item probabilities cannot replace its joint selection probability. Possible lists multiply rapidly, making full-slate weighting unstable. Estimators based on additive item-position contributions reduce this burden only by assuming away interactions among displayed items. That assumption is consequential precisely because list composition can change usefulness.

Measuring product benefit

A live comparison changes what people encounter. Specify both complete policies, the eligible population, assignment unit, actual exposure, primary outcome and observation horizon. Repeated visits by one person are not independent experimental units. Randomized assignment supports a causal comparison under the experiment's conditions; the broader design choices belong in Choose the live experiment. Preserve uncertainty and distinguish a practically useful effect from merely crossing a statistical threshold.

Guardrails are outcomes the change must not materially degrade, separate from its primary target. Shared inventory, creator attention or infrastructure can create interference, where one unit's outcome depends on other units' assignments. Cluster randomization groups interacting units; switchbacks alternate policies over time. Neither automatically eliminates spillovers, and both require an analysis suited to the design.

Measure beyond the surface receiving credit. YouTube's Learned Ranking Function study models value after a user leaves a recommendation slate, because activity can continue elsewhere on the platform. In a reported online ablation removing that baseline, the watch-page-attributed metric rose while the platform's primary metric fell. Local gains had not translated into a whole-product win. The reported primary metric should not be equated with independently established welfare or lifetime satisfaction.

Match follow-up to the claimed benefit. A click-only window ends before a purchase-return outcome matures. New navigation can temporarily slow established users, while a novel feature can prompt extra investigative clicking. Examine later visits and relevant cohorts rather than assuming the first response persists. Prespecify decision rules: repeatedly applying an ordinary fixed-horizon significance test until it passes does not preserve its original error guarantee.

Keeping recommendations useful

Shared models and serving choices

A shared recommendation foundation model supplies reusable learned representations or computation to multiple applications. It can consolidate repeated feature work without making every application share one objective. Netflix's 2025 talk describes thinner application models built above common user and content representations. Shared learning and application-specific decisions therefore coexist.

Netflix's November 2025 integration account reports three coexisting paths, reflecting different latency and technical requirements.
IntegrationReused componentDecision boundary
Embedding consumptionPublished user or item vectorsA downstream model uses representations that may predate the request.
Model subgraphShared computation inside an application modelThe application incorporates more of the model's computation.
Application fine-tuningShared parameters adapted to an applicationAdditional fitting changes behavior for the application's task.

Fine-tuning means further fitting model parameters; Post-training explains that mechanism. Moving work offline is a different choice. In YouTube's generative recommendation account, removing personalized prompt fields allowed video-to-video candidates for popular source videos to be precomputed and served by lookup. This avoided online generation for those candidates but removed user-specific conditioning from the table. A later stage may still personalize the final selection.

This schematic contrasts request-time generation using source-video and user context with precomputed candidates for popular source videos. Looking up candidates moves inference earlier; it does not establish identical quality, complete coverage, instantaneous refresh or zero serving cost.

Deployment value has always depended on more than predictive accuracy. In their April 2012 account, Xavier Amatriain and Justin Basilico explained that Netflix deployed adapted components from the 2007 Prize work but not the final 2009 ensemble. The additional accuracy did not justify the engineering effort, and streaming had changed the task from filling a DVD queue ahead of time to finding something to watch immediately. Streaming also supplied viewing feedback unavailable during mailed-DVD consumption. The lesson is to choose computation for the actual workflow, not to confuse nondeployment of one ensemble with rejection of the research.

Update the right boundary

Recommendation state changes on several clocks. Online computation uses current request information; offline computation prepares expensive results in batches; nearline computation reacts to events asynchronously between requests. A precomputed fallback can preserve response availability when online work cannot finish, but its freshness and personalization may differ. A technically successful response therefore does not establish a useful recommendation.

Changed informationResponsible updateVerification
Current session activityUpdate request or session state.The next selection receives the new context.
Availability or exclusionRefresh authoritative eligibility and dependent filters.Every serving path respects current restrictions.
Item representationRefresh compatible vectors and candidate index.Query encoding and stored vectors still share the scoring space.
Predictive relationshipEvaluate whether model refitting is needed.Later and live performance justify the change.
Outcome maturityResolve or retain pending labels according to the target window.Unfinished outcomes are not silently labeled negative.

Changing an encoder requires compatible preprocessing, item vectors, IDs, scoring conventions and index—not just matching dimensions. Preserve a compatible rollback bundle, while continuing to enforce current authority. Representation versioning develops the general contract. Netflix's November 2025 account separately describes model retraining, daily adaptation and batch representation publication; representation-space shifts require stabilization for downstream consumers. Those intervals are reported practices, not universal refresh targets.

Training–serving skew means features or prediction conditions differ between fitting and deployment. Compare sampled serving features with their training computations, and check performance as models age. Also distinguish a configured filter from propagated filter state: Amazon Personalize documents asynchronous updates, so its filter configuration is not proof of immediate exclusion. Outcome-connected observability supplies the general instrumentation principles.

Diagnose before enlarging the model. Inspect candidate-source contributions when useful inventory never reaches ranking; inspect targets when accurate predictions reward the wrong behavior; inspect list composition when recommendations repeat the same choice. Track new-item opportunities, exposure distribution, unwanted repeats, feature age and product outcomes for relevant groups. Then change the smallest responsible boundary, retain an executable baseline, and verify the complete policy under current conditions.

Open questions

  1. Learning from delayed outcomes while bounding exposure remains an operational challenge. Immediate signals arrive sooner than returns or sustained satisfaction, allowing many choices before their consequences mature. Progress would combine explicit exposure limits, pending-outcome handling and measured product guardrails in complete policy comparisons.

  2. Collection-level generation must reconcile sparse rewards with interactions among items. Successive decoding offers places to enforce diversity and business rules, but those possibilities do not establish valid constraints or better collections. Progress would demonstrate reliable constraint handling and whole-list benefit against simpler composition methods.

  3. Reusable user representations must remain responsive without destabilizing every downstream application. Longer histories, new interactions and model updates can alter representations in different ways. Progress would jointly measure current-task quality, stability across consumers and the cost of refreshing dependent state.

  4. Useful discovery is harder to measure than item difference. A varied list can be irrelevant, while rediscovering familiar material can be valuable. Progress would connect explicit discovery goals and user control with experienced value over repeated use, rather than substituting a diversity score for that experience.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

61 matching talks

TalkSpeakerEventYear
Hamed Firooz, Maziar SanjabiAI Engineer World's Fair 20252025
Rafal Wilinski, Vitor BaloccoAI Engineer World's Fair 20252025
Low Level Technicals of LLMs

Transcript reviewed

Daniel HanAI Engineer World's Fair 20242024
Jan SimlAI Engineer World's Fair 20252025
Jared JoselowitzAI Engineer World's Fair 20262026
Darius EmraniAI Engineer World's Fair 20252025
Ari HeljakkaAI Engineer World's Fair 20252025
Šimon PodhajskýAI Engineer Europe 20262026
Beyang LiuAI Engineer World's Fair 20242024
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Pablo CastroAI Engineer World's Fair 20242024
Jun Yu TanAI Engineer World's Fair 20252025
Bertrand CharpentierAI Engineer Europe 20262026
Tim AingeAI Engineer World's Fair 20262026
A Song of Types and Agents

Metadata candidate

Roberto StagiAI Engineer World's Fair 20262026
Boris Bogatin, Toufic BoubezAI Engineer Code 20252025
Thor Schaeff, Philipp SchmidAI Engineer Europe 20262026
Gergely Orosz, Simon EskildsenAI Engineer World's Fair 20262026
Cohere for VPs of AI

Metadata candidate

Vivek MuppallaAI Engineer World's Fair 20242024
Karina NguyenAI Engineer Summit 20252025
Mani KhanujaAI Engineer World's Fair 20252025
Ahmad AwaisAI Engineer Code 20252025
Arthur ObjartelAI Engineer Summit 20252025
Kevin HouAI Engineer World's Fair 20242024
Ishita DagaAI Engineer World's Fair 20262026
Benjamin FletcherAI Engineer World's Fair 20242024
Craig WattrusAI Engineer World's Fair 20252025
Emil EifremAI Engineer World's Fair 20242024
Iman MakaremiAI Engineer World's Fair 20252025
Rashi AgrawalAI Engineer World's Fair 20262026
Jaspreet SinghAI Engineer World's Fair 20252025
Hamel Husain, Greg CeccarelliAI Engineer Summit 20252025
Jeff Huber, Jason LiuAI Engineer World's Fair 20252025
Philipp KrennAI Engineer World's Fair 20252025
Justin McCartyAI Engineer World's Fair 20242024
Tom SmokerAI Engineer World's Fair 20252025
David KaramAI Engineer World's Fair 20252025
Shlok KhemaniAI Engineer World's Fair 20262026
Shafik Quoraishee, Joanne SongAI Engineer World's Fair 20262026
Mentoring the Machine

Metadata candidate

Eric HouAI Engineer World's Fair 20252025
Peter Werry, BrandonAI Engineer Europe 20262026
Rami AlhamadAI Engineer World's Fair 20252025
Atita Arora, Deanna EmeryAI Engineer World's Fair 20242024
Tengyu MaAI Engineer World's Fair 20252025
Anton TroynikovAI Engineer Summit 20232023
Shawn JanseparAI Engineer World's Fair 20242024
Scaling Compute on Context

Metadata candidate

Jack MorrisAI Engineer World's Fair 20262026
Arek BoruckiAI Engineer World's Fair 20262026
Sumit AgarwalAI Engineer World's Fair 20242024
Beyang LiuAI Engineer World's Fair 20252025
William LyonAI Engineer World's Fair 20252025
The Pipeline Is Dead

Metadata candidate

Iris ten TeijeAI Engineer World's Fair 20262026
Jonathan FernandesAI Engineer World's Fair 20252025
Frank LiuAI Engineer World's Fair 20252025
Sangwu LeeAI Engineer World's Fair 20262026
What the Best Agents Share

Metadata candidate

Mardu SwanepoelAI Engineer Europe 20262026
Andy TriedmanAI Engineer Summit 20252025
Devansh TandonAI Engineer World's Fair 20262026
Raghav SabooAI Engineer World's Fair 20262026
Yves Raimond, Jacqueline WoodAI Engineer World's Fair 20262026
Carlos SanchezAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
17 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
49 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. How YouTube recommendations work

    https://support.google.com/youtube/answer/16089387?hl=en

  2. How Zalando Uses Technology to Help Customers Find the Right Size

    https://corporate.zalando.com/en/technology/how-zalando-uses-technology-help-customers-find-right-size

  3. The effects of a personalized recommendation system on students’ high-stakes achievement scores: A field experiment

    https://files.eric.ed.gov/fulltext/ED615676.pdf

  4. Recommendation stages: eligibility, scoring and list constraints

    https://developers.google.com/machine-learning/recommendation/overview/types

  5. Off-policy evaluation for slate recommendation

    https://papers.neurips.cc/paper_files/paper/2017/file/5352696a9ca3397beb79f116f3a33991-Paper.pdf

  6. Personalization in the Era of LLMs

    https://www.youtube.com/watch?v=5YSJEP0HWzM&t=355

  7. Two-tower retrieval, approximation and compatible deployment artifacts

    https://www.tensorflow.org/recommenders/examples/efficient_serving

  8. GroupLens: An Open Architecture for Collaborative Filtering of Netnews

    https://pages.cpsc.ucalgary.ca/~saul/601.13/readings/p175-resnick.pdf

  9. Amazon.com Recommendations: Item-to-Item Collaborative Filtering

    https://www.cs.umd.edu/~samir/498/Amazon-Recommendations.pdf

  10. Matrix Factorization Techniques for Recommender Systems

    https://datajobs.com/data-science-repo/Recommender-Systems-%5BNetflix%5D.pdf

  11. The Netflix Prize

    https://www.cs.uic.edu/~liub/KDD-cup-2007/NetflixPrize-description.pdf

  12. YouTube training examples, sampled classes and watch-time ranking

    https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf

  13. Foundation Model for Personalized Recommendation

    https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39

  14. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    https://www.youtube.com/watch?v=2vlCqD6igVA&t=420

  15. Preparing item interaction data for training — Amazon Personalize

    https://docs.aws.amazon.com/personalize/latest/dg/interactions-datasets.html

  16. Modeling delayed conversion feedback

    https://wnzhang.net/share/rtb-papers/delayed-feedback.pdf

  17. One model to rule recommendations: Netflix's Big Bet

    https://www.youtube.com/watch?v=AbZ4IYGbfpQ&t=383

  18. Point-in-time historical feature joins

    https://raw.githubusercontent.com/feast-dev/feast/master/docs/getting-started/concepts/point-in-time-joins.md

  19. Content-based Filtering

    https://developers.google.com/machine-learning/recommendation/content-based/basics

  20. Item-based Collaborative Filtering Recommendation Algorithms — Memory-based Collaborative Filtering Algorithms

    https://www.ra.ethz.ch/CDstore/www10/papers/519/node7.html

  21. Item-based Collaborative Filtering Recommendation Algorithms — Item-based Collaborative Filtering Algorithm

    https://www.ra.ethz.ch/CDstore/www10/papers/519/node10.html

  22. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=2781

  23. Google Recommendation Systems: Matrix factorization

    https://developers.google.com/machine-learning/recommendation/collaborative/matrix

  24. Deep Learning, Chapter 2: Linear Algebra

    https://www.deeplearningbook.org/contents/linear_algebra.html

  25. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=406

  26. Anonymous session intent from ordered interactions

    https://arxiv.org/html/1511.06939

  27. Persistent preference, temporal drift and transient effects

    https://cseweb.ucsd.edu/classes/fa17/cse291-b/reading/p447-koren.pdf

  28. Personalization in the Era of LLMs: user steering

    https://www.youtube.com/watch?v=5YSJEP0HWzM

  29. 360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

    https://www.youtube.com/watch?v=U0S6CfzAY5c&t=476

  30. Re-identification risk in released recommendation histories

    https://arxiv.org/pdf/cs/0610105

  31. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=5532

  32. Relevance scoring in hybrid search using Reciprocal Rank Fusion — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking

  33. Filtering recommendations and user segments — Amazon Personalize

    https://docs.aws.amazon.com/personalize/latest/dg/filter.html

  34. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=3695

  35. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=3764

  36. Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

    https://www.youtube.com/watch?v=2DyHW23L6Cs&t=3868

  37. Personalization in the Era of LLMs

    https://www.youtube.com/watch?v=5YSJEP0HWzM&t=548

  38. Pointwise, pairwise and listwise learning to rank

    https://dmice.ohsu.edu/bedricks/courses/cs635_spring_2017/pdf/liu_2009.pdf

  39. BPR: Bayesian Personalized Ranking from Implicit Feedback

    https://www.cs.mcgill.ca/~uai2009/papers/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf

  40. Logistic regression: features, parameters and conditional probabilities

    https://developers.google.com/machine-learning/crash-course/logistic-regression/sigmoid-function

  41. Evaluate correctness, calibration, robustness and efficiency separately

    https://arxiv.org/pdf/2211.09110

  42. Learned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfaction

    https://arxiv.org/html/2408.06512

  43. What We Learned from Using LLMs in Pinterest

    https://www.youtube.com/watch?v=XdAWgO11zuk&t=895

  44. Temporal protocols and distinct recommendation-quality dimensions

    https://www.researchgate.net/publication/226264572_Evaluating_Recommendation_Systems

  45. What Is Serendipity? An Interview Study to Conceptualize Experienced Serendipity in Recommender Systems

    https://imec-publications.be/server/api/core/bitstreams/d53afde9-0a02-4872-a8bf-5861c0e69281/content

  46. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries

    https://www.researchgate.net/publication/243774776_The_use_of_mmr_and_diversity-based_reranking_in_document_reranking_and_summarization

  47. Learning a Personalized Homepage

    https://netflixtechblog.com/learning-a-personalized-homepage-aa8ec670359a

  48. Provider exposure as a ranking objective

    https://arxiv.org/pdf/1802.07281

  49. Unbiased Learning-to-Rank with Biased Feedback

    https://www.cs.cornell.edu/people/tj/publications/joachims_etal_17a.pdf

  50. Recommendations as Treatments: Debiasing Learning and Evaluation

    https://arxiv.org/html/1602.05352

  51. Algorithmic confounding and simulated homogenization

    https://arxiv.org/pdf/1710.11214

  52. Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms

    https://arxiv.org/html/1003.5956

  53. Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix

    https://arxiv.org/html/2608.21274v1

  54. Strategic manipulation through fabricated rating profiles

    https://files.grouplens.org/papers/p333-lam.pdf

  55. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    https://www.youtube.com/watch?v=2vlCqD6igVA&t=96

  56. Learning new-user preferences through informative rating requests

    https://www.kdd.org/exploration_files/WebKDD08-Al-Rashid.pdf

  57. Rules of ML: baselines, feature consistency and freshness monitoring

    https://developers.google.com/machine-learning/guides/rules-of-ml

  58. Contextual bandits: reward estimation plus exploration

    https://arxiv.org/html/1003.0146

  59. On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples

    https://www.scribd.com/document/887851455/RL-Thompson-1933-on-the-Likelihood-That-One-Unknown-Probability-Exceeds-Another-in-View-of-the-Evidence-of-Two-Samples

  60. A Tutorial on Thompson Sampling

    https://web.stanford.edu/~bvr/pubs/TS_Tutorial_FnT.pdf

  61. Contextual Bandit Content Personalization — VowpalWabbit 9.1.0

    https://vowpalwabbit.org/docs/vowpal_wabbit/python/9.1.0/tutorials/python_Simulating_a_news_personalization_scenario_using_Contextual_Bandits.html

  62. Guardrails and interference in online experiments

    https://arxiv.org/pdf/2212.11366

  63. Leveraging Unlabeled Data to Scale Blocking for Record Linkage

    https://www.ijcai.org/Proceedings/11/Papers/369.pdf

  64. Splink evaluation tutorial: labeled pairs and threshold selection

    https://moj-analytical-services.github.io/splink/demos/tutorials/07_Evaluation.html

  65. Recall@K: relevant-item denominator

    https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/RecallMetric

  66. NDCG@K: gains, discounts and ideal normalization

    https://github.com/tensorflow/ranking/blob/v0.5.3/tensorflow_ranking/python/keras/metrics.py

  67. Evaluation metrics under sampled candidates

    https://arxiv.org/pdf/1912.02263

  68. Randomized policy comparison and experimental uncertainty

    https://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTheWebSurvey.pdf

  69. Controlled experiments on the web: survey and practical guide

    https://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTheWebSurvey.pdf

  70. One model to rule recommendations: Netflix's Big Bet

    https://www.youtube.com/watch?v=AbZ4IYGbfpQ&t=193

  71. Integrating Netflix’s Foundation Model into Personalization Applications

    https://netflixtechblog.medium.com/integrating-netflixs-foundation-model-into-personalization-applications-cf176b5860eb

  72. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    https://www.youtube.com/watch?v=LxQsQ3vZDqo&t=629

  73. Netflix Recommendations: Beyond the 5 stars (Part 1)

    https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429

  74. Netflix architecture: online, nearline and fallback computation

    https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8

  75. Turning Fails into Features: Zapier’s Hard-Won Eval Lessons

    https://www.youtube.com/watch?v=blrovBxxN9o&t=930

  76. One model to rule recommendations: Netflix's Big Bet

    https://www.youtube.com/watch?v=AbZ4IYGbfpQ&t=1054

  77. One model to rule recommendations: Netflix's Big Bet

    https://www.youtube.com/watch?v=AbZ4IYGbfpQ&t=539

  78. Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

    https://www.youtube.com/watch?v=2vlCqD6igVA&t=927

  79. Personalization in the Era of LLMs

    https://www.youtube.com/watch?v=5YSJEP0HWzM&t=1048

  80. One model to rule recommendations: Netflix's Big Bet

    https://www.youtube.com/watch?v=AbZ4IYGbfpQ&t=696

  81. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    https://www.youtube.com/watch?v=LxQsQ3vZDqo&t=602