AI Engineer World's Fair 2025
What We Learned from Using LLMs in Pinterest
Loading slides…
Read the talk
What We Learned from Using LLMs in Pinterest
Pinterest combines joint query–Pin scoring, image captions and behavioral annotations with a distilled student that serves search and supplies reusable semantic embeddings.
From a talk by Mukuntha Narayanan and Han Wang
Before you start: Familiarity with embeddings, classification and the distinction between model training and inference will help with the architecture and serving details.
Which Pins answer a search query?
When someone types a query into Pinterest, which images will satisfy their information need? Search sits alongside home feed and related Pins as one of Pinterest’s three discovery surfaces. Han Wang and Mukuntha Narayanan, machine learning engineers on the Search Relevance team, focus on improving that query-driven experience.
Han reports more than six billion searches per month across billions of Pins, supporting over 45 languages and reaching users in more than 100 countries. Those searches span recipes, home decor, travel, fashion and other domains, so relevance modeling must work across both subjects and languages.
The search pipeline proceeds through query understanding → retrieval → reranking → blending, then produces a search feed. Semantic relevance is modeled during reranking: given a query and a candidate Pin, a classifier predicts one of five relevance levels, ranging from most relevant to most irrelevant. This is the specific task the LLM must improve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Score the query and Pin together
The relevance model concatenates the query and the Pin’s text, then passes them through an LLM. This cross-encoder lets the model capture interactions between the query and candidate content inside the encoder. An MLP consumes the resulting embedding and produces a five-dimensional vector corresponding to the relevance levels. Pinterest fine-tunes open-source language models on internal data to adapt this classifier to its content.
The baseline uses SearchSage, Pinterest’s in-house query and content embeddings. Across the tested models, Han reports stronger relevance prediction with more advanced and larger LLMs. Han reports that Llama 3 8B improves relevance prediction by approximately 12% over multilingual BERT base and 20% over SearchSage. The companion paper identifies the matching comparison as relative improvement in five-class accuracy, rather than percentage-point gains, with Pin text features and maximum input length held fixed across models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn images and user actions into text
A text-based relevance model needs a useful textual representation of an image-centered Pin. Pinterest builds that representation from five sources:
| Text source | Information supplied |
|---|---|
| Pin title | The Pin’s title text |
| Pin description | Its accompanying description |
| VLM-generated caption | Content extracted from the image |
| Saved-to board titles | How users organize the Pin |
| High-engagement search queries | Queries that lead users to engage with it |
The last two sources turn user actions into content annotations. Saving a Pin to a named board gives it context beyond its own description; queries associated with strong engagement provide another view of what people find it useful for. Together, the five sources give the language model a more comprehensive representation.
The ablation starts with the VLM caption alone, which Han describes as a strong baseline. Adding the other text features sequentially improves relevance prediction, including gains from the board titles and engaged queries. The useful result is not simply that more text helps: image-derived descriptions and behavior-derived annotations each contribute information that the original Pin text may lack.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the teacher to expand supervision
An accurate cross-encoder still leaves a production problem: how can Pinterest afford to apply its knowledge at search scale? Mukuntha introduces knowledge distillation into a smaller student. The student predicts the same five relevance levels, but learns from the teacher’s five-dimensional soft scores rather than only a single winning class.
A soft target preserves how the teacher distributes probability across relevance levels. The following Python loss illustrates training a student against such a distribution; teacher_probs contains one five-level distribution per query–Pin pair, while student_logits contains the student’s unnormalized predictions:
python
import torch
import torch.nn.functional as F
def soft_relevance_loss(
student_logits: torch.Tensor,
teacher_probs: torch.Tensor,
) -> torch.Tensor:
if student_logits.shape != teacher_probs.shape:
raise ValueError("Student and teacher shapes must match")
if student_logits.ndim != 2 or student_logits.shape[1] != 5:
raise ValueError("Expected tensors with shape [batch, 5]")
log_probs = F.log_softmax(student_logits, dim=-1)
targets = teacher_probs.detach()
return -(targets * log_probs).sum(dim=-1).mean()
This expresses the soft-supervision mechanism; the talk does not specify its exact training loss.
The semi-supervised data pipeline expands the reach of a relatively small human-labeled dataset:
- Train the teacher. Annotators trained for specific segments supply human labels. Pinterest fine-tunes a multilingual teacher using features that can apply across domains.
- Sample daily search logs. Sampling captures trending queries and fresh Pins, along with traffic from many languages and countries. Most human labels originate in the US, while only a small subset of this global traffic does.
- Label the sampled pairs. The teacher supplies relevance scores across the domains and markets where it produces useful labels.
- Train and serve the student. The expanded dataset transfers the teacher’s relevance judgments into the model that runs online.
Mukuntha reports approximately 100× expansion of the training data through teacher labeling. Daily sampling matters because the student’s supervision can cover current search behavior and content, not just the narrower population represented by the original human labels.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate encoding from online scoring
The student still contains language models, but it uses a bi-encoder: the Pin and query are encoded separately, without cross interactions inside their encoders. It also consumes SearchSage features, GraphSAGE embeddings, OmniSage and other embedding features. Query–Pin text-match statistics such as BM25 add signals that have historically worked well for Pinterest relevance prediction. Separate language encoding therefore does not mean that the final score is merely an embedding similarity.
The serving advantage follows from separating the two inputs:
| Component | Computation strategy |
|---|---|
| Pin embedding | Infer offline across billions of Pins |
| Pin refresh | Recompute when input features meaningfully change |
| Query embedding | Infer online from a short query |
| Repeated query | Reuse its cached embedding |
The Pin encoder uses predominantly the same text features as the teacher, helping transfer its knowledge. Once computed, a Pin’s representation can be reused across searches; its encoding is not part of the work triggered by an individual search request.
Mukuntha reports query-embedding latency of a few milliseconds for short search queries. Repeated queries provide another saving: Mukuntha reports an approximately 85% query-embedding cache hit rate. These figures describe the query-encoding path and its cache, not end-to-end search latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure relevance and fulfillment across markets
The online evaluation examines relevance through nDCG and precision at eight in selected US, Germany and France segments. These are relevance measurements rather than engagement measures; the companion paper uses graded labels for precision at eight and an all-highest-relevance ideal for nDCG. Mukuntha reports international relevance gains despite starting this experiment with limited, predominantly US human-labeled data.
Pinterest also measures search fulfillment, which tracks fulfilling engagement actions on search. Mukuntha reports that search fulfillment also improves outside the US. The multilingual teacher makes the limited initial supervision useful across a broader set of countries and domains, while the student makes those learned judgments practical to serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse the representations beyond search
The production student produces more than relevance scores. Its Pin and query embeddings are already available as semantic representations, so other systems can reuse them without building a separate representation model for each application. Pinterest also uses the model to represent boards from their titles and related text. Mukuntha attributes the representations’ richness to relevance tuning, distillation from the teacher and the language models within the student.
Mukuntha reports improvements from adding these representations to related Pins, home feed and other surfaces. The applications slide extends that reuse across retrieval, ranking and content moderation. Each displayed result belongs to its associated application and metric; they are separate downstream uses of the embeddings, rather than one aggregate search result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the model and locate its serving role
Asked how Pinterest chose which open-source LLM to fine-tune, Han describes an empirical selection process: experiment with multiple language models, compare their measured performance and select the strongest performer. The answer supplies no additional selection rule beyond those experiments.
A second question separates two roles that are easy to conflate: the LLM that produces training labels and the model that scores a live search. Mukuntha clarifies that the teacher distills its judgments into a five-level relevance student. Candidate generators and early ranking run first; the student scores relevance later, during reranking immediately before blending the results into a feed. The teacher’s influence reaches serving through the trained student.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Broader coverage with captions and one multilingual model
The next question asks what drove adoption: did LLMs enable new search features, or improve capabilities that the existing system already had? Mukuntha emphasizes the ability to expand relevance measurement and labeling beyond a limited set of markets. VLM captions improve the content representation, while multilingual models supply synthetic relevance data for additional markets. The benefit described is broader relevance coverage, rather than a new user-facing search interaction.
Asked whether the embedding model inherently combines text and images, Mukuntha says direct multimodality is under exploration. In the approach described here, visual captions capture image content effectively for many applications, supported by Pinterest’s in-house captioning models. The distinction matters: images contribute through their textual descriptions, without requiring the relevance encoder itself to ingest pixels.
The final question asks whether international improvements require separate models for individual languages or a common model. Han confirms that all languages use the same model, attributing cross-language transfer to the multilingual language model. That answer establishes shared modeling across languages, without claiming that every feature is identical in every market.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Pinterest's original account of SearchSage query embeddings for search retrieval and ranking.
Pinterest's framework combining graph, content and user-sequence signals into reusable representations.
Original research on learning representations that generalize to previously unseen graph nodes.
Further reading
Original research detailing relevance classification, distillation and offline and online evaluations.
The speakers' companion explanation of text features, teacher training and production relevance scoring.
- OmniSearchSagePaper
Joint query, Pin and product embeddings trained through multiple tasks for Pinterest Search.
Official research implementation with source code, tests and dependency requirements.
Full transcript
Loading transcript…