AI Engineer World's Fair 2025

What We Learned from Using LLMs in Pinterest

Mukuntha Narayanan· Machine Learning Engineer, Relevance & Query Understanding, PinterestHan Wang· Machine Learning Engineer, Relevance & Query Understanding, Pinterest18:13

Loading slides…

Read the talk

What We Learned from Using LLMs in Pinterest

Pinterest combines joint query–Pin scoring, image captions and behavioral annotations with a distilled student that serves search and supplies reusable semantic embeddings.

From a talk by Mukuntha Narayanan and Han Wang

Before you start: Familiarity with embeddings, classification and the distinction between model training and inference will help with the architecture and serving details.

Which Pins answer a search query?

When someone types a query into Pinterest, which images will satisfy their information need? Search sits alongside home feed and related Pins as one of Pinterest’s three discovery surfaces. Han Wang and Mukuntha Narayanan, machine learning engineers on the Search Relevance team, focus on improving that query-driven experience.

Three mobile interface examples show Home Feed, Related Pins, and Search, with image grids beneath each heading.Open full source frame
Pinterest’s three discovery surfaces: Home Feed, Related Pins, and Search.

Han reports more than six billion searches per month across billions of Pins, supporting over 45 languages and reaching users in more than 100 countries. Those searches span recipes, home decor, travel, fashion and other domains, so relevance modeling must work across both subjects and languages.

The search pipeline proceeds through query understanding → retrieval → reranking → blending, then produces a search feed. Semantic relevance is modeled during reranking: given a query and a candidate Pin, a classifier predicts one of five relevance levels, ranging from most relevant to most irrelevant. This is the specific task the LLM must improve.

0:270:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:27 · section reference included

Score the query and Pin together

The relevance model concatenates the query and the Pin’s text, then passes them through an LLM. This cross-encoder lets the model capture interactions between the query and candidate content inside the encoder. An MLP consumes the resulting embedding and produces a five-dimensional vector corresponding to the relevance levels. Pinterest fine-tunes open-source language models on internal data to adapt this classifier to its content.

Model Architecture slide shows [CLS], Query, [SEP], Pin, and Text entering an Encoder LLM, followed by an MLP and five relevance circles. Bullets describe multiclass classification and fine-tuning on Pinterest data.Open full source frame
A cross-encoder feeds joint query and Pin text representations into an MLP for five-scale relevance prediction.

The baseline uses SearchSage, Pinterest’s in-house query and content embeddings. Across the tested models, Han reports stronger relevance prediction with more advanced and larger LLMs. Han reports that Llama 3 8B improves relevance prediction by approximately 12% over multilingual BERT base and 20% over SearchSage. The companion paper identifies the matching comparison as relative improvement in five-class accuracy, rather than percentage-point gains, with Pin text features and maximum input length held fixed across models.

3:173:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:17 · section reference included

Turn images and user actions into text

A text-based relevance model needs a useful textual representation of an image-centered Pin. Pinterest builds that representation from five sources:

Text sourceInformation supplied
Pin titleThe Pin’s title text
Pin descriptionIts accompanying description
VLM-generated captionContent extracted from the image
Saved-to board titlesHow users organize the Pin
High-engagement search queriesQueries that lead users to engage with it

The last two sources turn user actions into content annotations. Saving a Pin to a named board gives it context beyond its own description; queries associated with strong engagement provide another view of what people find it useful for. Together, the five sources give the language model a more comprehensive representation.

The ablation starts with the VLM caption alone, which Han describes as a strong baseline. Adding the other text features sequentially improves relevance prediction, including gains from the board titles and engaged queries. The useful result is not simply that more text helps: image-derived descriptions and behavior-derived annotations each contribute information that the original Pin text may lack.

5:045:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:04 · section reference included

Use the teacher to expand supervision

An accurate cross-encoder still leaves a production problem: how can Pinterest afford to apply its knowledge at search scale? Mukuntha introduces knowledge distillation into a smaller student. The student predicts the same five relevance levels, but learns from the teacher’s five-dimensional soft scores rather than only a single winning class.

A soft target preserves how the teacher distributes probability across relevance levels. The following Python loss illustrates training a student against such a distribution; teacher_probs contains one five-level distribution per query–Pin pair, while student_logits contains the student’s unnormalized predictions:

python

import torch
import torch.nn.functional as F


def soft_relevance_loss(
    student_logits: torch.Tensor,
    teacher_probs: torch.Tensor,
) -> torch.Tensor:
    if student_logits.shape != teacher_probs.shape:
        raise ValueError("Student and teacher shapes must match")
    if student_logits.ndim != 2 or student_logits.shape[1] != 5:
        raise ValueError("Expected tensors with shape [batch, 5]")

    log_probs = F.log_softmax(student_logits, dim=-1)
    targets = teacher_probs.detach()
    return -(targets * log_probs).sum(dim=-1).mean()

This expresses the soft-supervision mechanism; the talk does not specify its exact training loss.

The semi-supervised data pipeline expands the reach of a relatively small human-labeled dataset:

  1. Train the teacher. Annotators trained for specific segments supply human labels. Pinterest fine-tunes a multilingual teacher using features that can apply across domains.
  2. Sample daily search logs. Sampling captures trending queries and fresh Pins, along with traffic from many languages and countries. Most human labels originate in the US, while only a small subset of this global traffic does.
  3. Label the sampled pairs. The teacher supplies relevance scores across the domains and markets where it produces useful labels.
  4. Train and serve the student. The expanded dataset transfers the teacher’s relevance judgments into the model that runs online.

Mukuntha reports approximately 100× expansion of the training data through teacher labeling. Daily sampling matters because the student’s supervision can cover current search behavior and content, not just the narrower population represented by the original human labels.

7:147:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:14 · section reference included

Separate encoding from online scoring

The student still contains language models, but it uses a bi-encoder: the Pin and query are encoded separately, without cross interactions inside their encoders. It also consumes SearchSage features, GraphSAGE embeddings, OmniSage and other embedding features. Query–Pin text-match statistics such as BM25 add signals that have historically worked well for Pinterest relevance prediction. Separate language encoding therefore does not mean that the final score is merely an embedding similarity.

The serving advantage follows from separating the two inputs:

ComponentComputation strategy
Pin embeddingInfer offline across billions of Pins
Pin refreshRecompute when input features meaningfully change
Query embeddingInfer online from a short query
Repeated queryReuse its cached embedding

The Pin encoder uses predominantly the same text features as the teacher, helping transfer its knowledge. Once computed, a Pin’s representation can be reused across searches; its encoding is not part of the work triggered by an individual search request.

Mukuntha reports query-embedding latency of a few milliseconds for short search queries. Repeated queries provide another saving: Mukuntha reports an approximately 85% query-embedding cache hit rate. These figures describe the query-encoding path and its cache, not end-to-end search latency.

9:129:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:12 · section reference included

Measure relevance and fulfillment across markets

The online evaluation examines relevance through nDCG and precision at eight in selected US, Germany and France segments. These are relevance measurements rather than engagement measures; the companion paper uses graded labels for precision at eight and an all-highest-relevance ideal for nDCG. Mukuntha reports international relevance gains despite starting this experiment with limited, predominantly US human-labeled data.

Pinterest also measures search fulfillment, which tracks fulfilling engagement actions on search. Mukuntha reports that search fulfillment also improves outside the US. The multilingual teacher makes the limited initial supervision useful across a broader set of countries and domains, while the student makes those learned judgments practical to serve.

11:1711:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:17 · section reference included

Reuse the representations beyond search

The production student produces more than relevance scores. Its Pin and query embeddings are already available as semantic representations, so other systems can reuse them without building a separate representation model for each application. Pinterest also uses the model to represent boards from their titles and related text. Mukuntha attributes the representations’ richness to relevance tuning, distillation from the teacher and the language models within the student.

Mukuntha reports improvements from adding these representations to related Pins, home feed and other surfaces. The applications slide extends that reuse across retrieval, ranking and content moderation. Each displayed result belongs to its associated application and metric; they are separate downstream uses of the embeddings, rather than one aggregate search result.

LLM Embedding Applications slide lists +1.1% Pin Shares, +0.85% Repins, +16% Recall@10, +1.4% Repins, 2.7x improved content moderation precision, and +0.12% sessions longer than five minutes, with their associated applications.Open full source frame
Reported applications of LLM embeddings across retrieval, ranking, and content moderation.
12:1012:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:10 · section reference included

Choose the model and locate its serving role

Asked how Pinterest chose which open-source LLM to fine-tune, Han describes an empirical selection process: experiment with multiple language models, compare their measured performance and select the strongest performer. The answer supplies no additional selection rule beyond those experiments.

A second question separates two roles that are easy to conflate: the LLM that produces training labels and the model that scores a live search. Mukuntha clarifies that the teacher distills its judgments into a five-level relevance student. Candidate generators and early ranking run first; the student scores relevance later, during reranking immediately before blending the results into a feed. The teacher’s influence reaches serving through the trained student.

14:0314:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:03 · section reference included

Broader coverage with captions and one multilingual model

The next question asks what drove adoption: did LLMs enable new search features, or improve capabilities that the existing system already had? Mukuntha emphasizes the ability to expand relevance measurement and labeling beyond a limited set of markets. VLM captions improve the content representation, while multilingual models supply synthetic relevance data for additional markets. The benefit described is broader relevance coverage, rather than a new user-facing search interaction.

Asked whether the embedding model inherently combines text and images, Mukuntha says direct multimodality is under exploration. In the approach described here, visual captions capture image content effectively for many applications, supported by Pinterest’s in-house captioning models. The distinction matters: images contribute through their textual descriptions, without requiring the relevance encoder itself to ingest pixels.

The final question asks whether international improvements require separate models for individual languages or a common model. Han confirms that all languages use the same model, attributing cross-language transfer to the multilingual language model. That answer establishes shared modeling across languages, without claiming that every feature is identical in every market.

15:3415:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:34 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…