# What We Learned from Using LLMs in Pinterest

Mukuntha Narayanan, Pinterest · Han Wang, Pinterest

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/XdAWgO11zuk-what-we-learned-from-using-llms) · [Video](https://www.youtube.com/watch?v=XdAWgO11zuk)

## What We Learned from Using LLMs in Pinterest

Pinterest combines joint query–Pin scoring, image captions and behavioral annotations with a distilled student that serves search and supplies reusable semantic embeddings.

Familiarity with embeddings, classification and the distinction between model training and inference will help with the architecture and serving details.

### Which Pins answer a search query?

[0:27](https://www.youtube.com/watch?v=XdAWgO11zuk&t=27s)

When someone types a query into Pinterest, which images will satisfy their information need? Search sits alongside home feed and related Pins as one of Pinterest’s three discovery surfaces. Han Wang and Mukuntha Narayanan, machine learning engineers on the Search Relevance team, focus on improving that query-driven experience.

![Three mobile interface examples show Home Feed, Related Pins, and Search, with image grids beneath each heading.](https://ai.engineer/api/talk-essay-media/f108b614007b6a2bfe60225536256661d437b010cbcc335de6de28fb7511167d)

Pinterest’s three discovery surfaces: Home Feed, Related Pins, and Search.

Han reports more than six billion searches per month across billions of Pins, supporting over 45 languages and reaching users in more than 100 countries. Those searches span recipes, home decor, travel, fashion and other domains, so relevance modeling must work across both subjects and languages.

The search pipeline proceeds through **query understanding → retrieval → reranking → blending**, then produces a search feed. Semantic relevance is modeled during reranking: given a query and a candidate Pin, a classifier predicts one of five relevance levels, ranging from most relevant to most irrelevant. This is the specific task the LLM must improve.

### Score the query and Pin together

[3:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=197s)

The relevance model concatenates the query and the Pin’s text, then passes them through an LLM. This **cross-encoder** lets the model capture interactions between the query and candidate content inside the encoder. An MLP consumes the resulting embedding and produces a five-dimensional vector corresponding to the relevance levels. Pinterest fine-tunes open-source language models on internal data to adapt this classifier to its content.

![Model Architecture slide shows CLS, Query, SEP, Pin, and Text entering an Encoder LLM, followed by an MLP and five relevance circles. Bullets describe multiclass classification and fine-tuning on Pinterest data.](https://ai.engineer/api/talk-essay-media/0fc865c93595559b99dd7ebed2b057ad580a995f18596c4be740a452f4a3639e)

A cross-encoder feeds joint query and Pin text representations into an MLP for five-scale relevance prediction.

The baseline uses **[SearchSage](https://medium.com/pinterest-engineering/searchsage-learning-search-query-representations-at-pinterest-654f2bb887fc)**, Pinterest’s in-house query and content embeddings. Across the tested models, Han reports stronger relevance prediction with more advanced and larger LLMs. Han reports that **Llama 3 8B** improves relevance prediction by approximately 12% over **multilingual BERT base** and 20% over SearchSage. The [companion paper](https://arxiv.org/html/2410.17152v1) identifies the matching comparison as relative improvement in five-class accuracy, rather than percentage-point gains, with Pin text features and maximum input length held fixed across models.

### Turn images and user actions into text

[5:04](https://www.youtube.com/watch?v=XdAWgO11zuk&t=304s)

A text-based relevance model needs a useful textual representation of an image-centered Pin. Pinterest builds that representation from five sources:

| Text source | Information supplied |
| --- | --- |
| Pin title | The Pin’s title text |
| Pin description | Its accompanying description |
| VLM-generated caption | Content extracted from the image |
| Saved-to board titles | How users organize the Pin |
| High-engagement search queries | Queries that lead users to engage with it |

The last two sources turn **user actions into content annotations**. Saving a Pin to a named board gives it context beyond its own description; queries associated with strong engagement provide another view of what people find it useful for. Together, the five sources give the language model a more comprehensive representation.

The ablation starts with the VLM caption alone, which Han describes as a strong baseline. Adding the other text features sequentially improves relevance prediction, including gains from the board titles and engaged queries. The useful result is not simply that more text helps: image-derived descriptions and behavior-derived annotations each contribute information that the original Pin text may lack.

### Use the teacher to expand supervision

[7:14](https://www.youtube.com/watch?v=XdAWgO11zuk&t=434s)

An accurate cross-encoder still leaves a production problem: how can Pinterest afford to apply its knowledge at search scale? Mukuntha introduces **knowledge distillation** into a smaller student. The student predicts the same five relevance levels, but learns from the teacher’s five-dimensional soft scores rather than only a single winning class.

A soft target preserves how the teacher distributes probability across relevance levels. The following Python loss illustrates training a student against such a distribution; `teacher_probs` contains one five-level distribution per query–Pin pair, while `student_logits` contains the student’s unnormalized predictions:

```python
import torch
import torch.nn.functional as F


def soft_relevance_loss(
    student_logits: torch.Tensor,
    teacher_probs: torch.Tensor,
) -> torch.Tensor:
    if student_logits.shape != teacher_probs.shape:
        raise ValueError("Student and teacher shapes must match")
    if student_logits.ndim != 2 or student_logits.shape[1] != 5:
        raise ValueError("Expected tensors with shape [batch, 5]")

    log_probs = F.log_softmax(student_logits, dim=-1)
    targets = teacher_probs.detach()
    return -(targets * log_probs).sum(dim=-1).mean()
```

This expresses the soft-supervision mechanism; the talk does not specify its exact training loss.

The semi-supervised data pipeline expands the reach of a relatively small human-labeled dataset:

1. **Train the teacher.** Annotators trained for specific segments supply human labels. Pinterest fine-tunes a multilingual teacher using features that can apply across domains.
2. **Sample daily search logs.** Sampling captures trending queries and fresh Pins, along with traffic from many languages and countries. Most human labels originate in the US, while only a small subset of this global traffic does.
3. **Label the sampled pairs.** The teacher supplies relevance scores across the domains and markets where it produces useful labels.
4. **Train and serve the student.** The expanded dataset transfers the teacher’s relevance judgments into the model that runs online.

Mukuntha reports approximately 100× expansion of the training data through teacher labeling. Daily sampling matters because the student’s supervision can cover current search behavior and content, not just the narrower population represented by the original human labels.

### Separate encoding from online scoring

[9:12](https://www.youtube.com/watch?v=XdAWgO11zuk&t=552s)

The student still contains language models, but it uses a **bi-encoder**: the Pin and query are encoded separately, without cross interactions inside their encoders. It also consumes SearchSage features, **[GraphSAGE](https://arxiv.org/abs/1706.02216)** embeddings, **[OmniSage](https://arxiv.org/abs/2504.17811)** and other embedding features. Query–Pin text-match statistics such as **BM25** add signals that have historically worked well for Pinterest relevance prediction. Separate language encoding therefore does not mean that the final score is merely an embedding similarity.

The serving advantage follows from separating the two inputs:

| Component | Computation strategy |
| --- | --- |
| Pin embedding | Infer offline across billions of Pins |
| Pin refresh | Recompute when input features meaningfully change |
| Query embedding | Infer online from a short query |
| Repeated query | Reuse its cached embedding |

The Pin encoder uses predominantly the same text features as the teacher, helping transfer its knowledge. Once computed, a Pin’s representation can be reused across searches; its encoding is not part of the work triggered by an individual search request.

Mukuntha reports query-embedding latency of a few milliseconds for short search queries. Repeated queries provide another saving: Mukuntha reports an approximately 85% query-embedding cache hit rate. These figures describe the query-encoding path and its cache, not end-to-end search latency.

### Measure relevance and fulfillment across markets

[11:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=677s)

The online evaluation examines relevance through nDCG and precision at eight in selected US, Germany and France segments. These are relevance measurements rather than engagement measures; the companion paper uses graded labels for precision at eight and an all-highest-relevance ideal for nDCG. Mukuntha reports international relevance gains despite starting this experiment with limited, predominantly US human-labeled data.

Pinterest also measures **search fulfillment**, which tracks fulfilling engagement actions on search. Mukuntha reports that search fulfillment also improves outside the US. The multilingual teacher makes the limited initial supervision useful across a broader set of countries and domains, while the student makes those learned judgments practical to serve.

### Reuse the representations beyond search

[12:10](https://www.youtube.com/watch?v=XdAWgO11zuk&t=730s)

The production student produces more than relevance scores. Its Pin and query embeddings are already available as semantic representations, so other systems can reuse them without building a separate representation model for each application. Pinterest also uses the model to represent boards from their titles and related text. Mukuntha attributes the representations’ richness to relevance tuning, distillation from the teacher and the language models within the student.

Mukuntha reports improvements from adding these representations to related Pins, home feed and other surfaces. The applications slide extends that reuse across retrieval, ranking and content moderation. Each displayed result belongs to its associated application and metric; they are separate downstream uses of the embeddings, rather than one aggregate search result.

![LLM Embedding Applications slide lists +1.1% Pin Shares, +0.85% Repins, +16% Recall@10, +1.4% Repins, 2.7x improved content moderation precision, and +0.12% sessions longer than five minutes, with their associated applications.](https://ai.engineer/api/talk-essay-media/fbfd567cf43f2c7a32299b961c5f1561af1d449706736e722fb4277323dc1b62)

Reported applications of LLM embeddings across retrieval, ranking, and content moderation.

### Choose the model and locate its serving role

[14:03](https://www.youtube.com/watch?v=XdAWgO11zuk&t=843s)

Asked how Pinterest chose which open-source LLM to fine-tune, Han describes an empirical selection process: experiment with multiple language models, compare their measured performance and select the strongest performer. The answer supplies no additional selection rule beyond those experiments.

A second question separates two roles that are easy to conflate: the LLM that produces training labels and the model that scores a live search. Mukuntha clarifies that the teacher distills its judgments into a five-level relevance student. Candidate generators and early ranking run first; the student scores relevance later, during reranking immediately before blending the results into a feed. The teacher’s influence reaches serving through the trained student.

### Broader coverage with captions and one multilingual model

[15:34](https://www.youtube.com/watch?v=XdAWgO11zuk&t=934s)

The next question asks what drove adoption: did LLMs enable new search features, or improve capabilities that the existing system already had? Mukuntha emphasizes the ability to expand relevance measurement and labeling beyond a limited set of markets. VLM captions improve the content representation, while multilingual models supply synthetic relevance data for additional markets. The benefit described is broader relevance coverage, rather than a new user-facing search interaction.

Asked whether the embedding model inherently combines text and images, Mukuntha says direct multimodality is under exploration. In the approach described here, visual captions capture image content effectively for many applications, supported by Pinterest’s in-house captioning models. The distinction matters: images contribute through their textual descriptions, without requiring the relevance encoder itself to ingest pixels.

The final question asks whether international improvements require separate models for individual languages or a common model. Han confirms that **all languages use the same model**, attributing cross-language transfer to the multilingual language model. That answer establishes shared modeling across languages, without claiming that every feature is identical in every market.

## Resources

- [Improving Pinterest Search Relevance Using Large Language Models](https://arxiv.org/html/2410.17152v1) — Original research detailing relevance classification, distillation and offline and online evaluations.
- [Pinterest's search relevance engineering walkthrough](https://medium.com/pinterest-engineering/improving-pinterest-search-relevance-using-large-language-models-4cd938d4e892) — The speakers' companion explanation of text features, teacher training and production relevance scoring.
- [SearchSage: Learning Search Query Representations at Pinterest](https://medium.com/pinterest-engineering/searchsage-learning-search-query-representations-at-pinterest-654f2bb887fc) — Pinterest's original account of SearchSage query embeddings for search retrieval and ranking.
- [OmniSearchSage](https://arxiv.org/abs/2404.16260) — Joint query, Pin and product embeddings trained through multiple tasks for Pinterest Search.
- [OmniSearchSage source code](https://github.com/pinterest/atg-research/tree/main/omnisearchsage) — Official research implementation with source code, tests and dependency requirements.
- [OmniSage](https://arxiv.org/abs/2504.17811) — Pinterest's framework combining graph, content and user-sequence signals into reusable representations.
- [GraphSAGE: Inductive Representation Learning on Large Graphs](https://arxiv.org/abs/1706.02216) — Original research on learning representations that generalize to previously unseen graph nodes.

## Slides

### What We Learned from Using LLMs in Pinterest Search

Han Wang and Mukuntha Narayanan, machine learning engineers on Pinterest’s Search Relevance team, share what they learned from integrating LLMs into Pinterest Search.

[0:26](https://www.youtube.com/watch?v=XdAWgO11zuk&t=26s) [Title slide naming Han Wang and Mukuntha Narayanan, alongside a collage of Pinterest images.](https://assets.ai.engineer/sites/aiecode2025/sha256/d32541c638a141b73c9986070ee8532cd5461309d0edf609a1daf7088aa397d6.webp)

What We Learned from Using LLMs in Pinterest Search

Han Wang
Mukuntha Narayanan

Interface text:
Save

### Pinterest discovery surfaces

Pinterest shows three discovery surfaces: the Home Feed, Related Pins, and Search. The Search example shows results for “Cat”; the talk focuses on improving search relevance with LLMs.

[0:34](https://www.youtube.com/watch?v=XdAWgO11zuk&t=34s) [Examples of Pinterest Home Feed, Related Pins, and Search screens.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef60250ec58a0c74101ff4e23818c50c40ed2f6820d30f6c74136334b5740c9f.webp)

Home Feed
Related Pins
Search

Interface text:
For you
Vacation outfits
Haircut inspo
Col[illegible]
Save
More to explore
Cat
Cute
Silly
Memes
Aes[illegible]
AIE
Microsoft
smol ai

### Pinterest search statistics

Pinterest reports over 6 billion monthly searches across billions of Pins, supporting 45+ languages and reaching 100+ countries. The monthly search figure cites global internal data from June 2023.

[1:12](https://www.youtube.com/watch?v=XdAWgO11zuk&t=72s) [Pinterest search statistics show over six billion monthly searches, billions of Pins, more than 45 languages, and over 100 countries.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e202a752d2d273165fe40c559182c5db6869142ae1d828bdc6a9953763afa66.webp)

Search Statistics

Search is one of the major discovery surface at Pinterest where Pinners come for find inspiration.

6B+
searches per month¹

Billions
Of Pins to search from

At Pinterest, search is remarkably global and multilingual.

45+
Languages with English, Spanish, Portuguese being the most frequently used

100+
Countries with majority from US, Brazil, Mexico, India, Indonesia

1. Pinterest Internal Data, Global, June 2023

### Search System Overview

Pinterest Search moves from query and user understanding through retrieval, reranking and blending to results. The talk focuses on semantic relevance modeling at the reranking stage.

[1:50](https://www.youtube.com/watch?v=XdAWgO11zuk&t=110s) [Diagram of query and user understanding, retrieval and lightweight scoring, reranking, blending, and search results.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d6c8e2063008e54d8ceef8b1b7c842f9ccbd9329d329c58cddb39b1c3fc29ca.webp)

Search System Overview

User performs a search

Query / User Understanding
Retrieval + Lightweight Scoring
Reranking
engagement
semantic relevance
Blending
1
2
3
Results
Pins + Modules

Interface text:
AIE
Microsoft
smol ai

[2:04](https://www.youtube.com/watch?v=XdAWgO11zuk&t=124s) [Diagram of search stages: query understanding, retrieval, reranking, blending, and results.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a16215f805f9abfb7ca0e29e74862e026c0be9f606c87919d716675f81a5538.webp)

Search System Overview

User performs a search

Query / User Understanding
Retrieval + Lightweight Scoring
Reranking
engagement
semantic relevance
Blending
1
2
3
Results
Pins + Modules

### Search Relevance Model

A classifier compares a search query with a Pin and predicts relevance on a five-point scale. The example pairs the query “cat” with a cat Pin.

[2:28](https://www.youtube.com/watch?v=XdAWgO11zuk&t=148s) [A cat query and Pin feed a relevance model that assigns one of five relevance levels.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c98e16d02132e25261c116f73ac12cd90815781ccac7d0e529b5e7722a75f4f.webp)

Search Relevance Model

search query
cat
+
pin

relevance model

5 - Excellent / Most Relevant
4 - Good / Relevant
3 - Complementary / Marginal
2- Bad / Not Relevant
1- Very Bad / Offensive

Interface text:
Save

### Agenda

Four lessons on Pinterest search relevance cover LLM predictions, content annotations, efficient serving through distillation, and relevance-tuned semantic representations.

[2:54](https://www.youtube.com/watch?v=XdAWgO11zuk&t=174s) [Four lessons cover relevance prediction, content annotations, knowledge distillation, and semantic representations.](https://assets.ai.engineer/sites/aiecode2025/sha256/55df26575c1bbf3e571af27d56402eb0864119e88f7106a9d0bf8e280897b90c.webp)

Agenda

1
Lesson 1
LLMs are good at relevance prediction

2
Lesson 2
VLM captions and user actions are useful content annotations

3
Lesson 3
Knowledge distillation is the key to efficient serving

4
Lesson 4
Relevance-tuning produces rich semantic representations

### Lesson 1

The first lesson focuses on using LLMs to predict relevance for Pinterest search.

[3:09](https://www.youtube.com/watch?v=XdAWgO11zuk&t=189s) [Lesson 1: LLMs are good at relevance prediction.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0a2ee3221253f4b759c596d130ac7573ec5d89f94343ea47f89d10e9c1d4747.webp)

Lesson 1
LLMs are good at relevance prediction

### Model Architecture

A cross-encoder combines query and Pin text into an embedding. An MLP maps it to five relevance levels; the open-source LLMs are fine-tuned on Pinterest data.

[3:16](https://www.youtube.com/watch?v=XdAWgO11zuk&t=196s) [A cross-encoder LLM processes query and Pin text, with an MLP producing five-scale relevance predictions.](https://assets.ai.engineer/sites/aiecode2025/sha256/a50275cc92388b722c9059f8f3f661c35dc485175be03e84798affcbcf4093a0.webp)

Model Architecture

• multi-class classification
• cross-encoder structure to better capture the interaction between query and Pin
• finetune open-source LLMs on Pinterest data

5-scale relevance
MLP
Encoder LLM
[CLS] Query [SEP] Pin Text

[3:23](https://www.youtube.com/watch?v=XdAWgO11zuk&t=203s) [A cross-encoder combines query and Pin text, then an MLP predicts five relevance levels.](https://assets.ai.engineer/sites/aiecode2025/sha256/93bc4917bf972f8ce22e07a5bca7788bb7838e3ff667f2d0c3303f6a66e08adf.webp)

Model Architecture

• multi-class classification
• cross-encoder structure to better capture the interaction between query and Pin
• finetune open-source LLMs on Pinterest data

5-scale relevance
MLP
Encoder LLM
[CLS] Query [SEP] Pin Text

Interface text:
AIE
Microsoft
smol ai

### LLM Performance

Accuracy on five-scale relevance prediction rises from 0.503 for SearchSage to 0.602 for Llama-3-8B. AUROC also improves across the models shown at all three thresholds.

[4:06](https://www.youtube.com/watch?v=XdAWgO11zuk&t=246s) [A table compares relevance prediction accuracy and AUROC across language models and the SearchSage baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/8eb428a8fbc0f73836109967dc475f36c5ce96c9ec0db410bf46bc5b678afdce.webp)

LLM Performance

• language models offer additional improvements over our in-house content and query embedding SearchSAGE
• more sophisticated language models and larger model sizes consistently enhance the relevance prediction performance

Table 1: Comparisons of different language models on 5-scale relevance prediction. The AUROC metrics are reported for binarized labels with thresholds 3, 4, and 5.

Model | Accuracy | AUROC 3+/4+/5+
SearchSage | 0.503 | 0.878/0.845/0.826
mBERT_base | 0.535 | 0.887/0.864/0.861
T5_base | 0.569 | 0.909/0.884/0.886
mDeBERTaV3_base | 0.580 | 0.917/0.892/0.895
XLM-RoBERTa_large | 0.588 | 0.919/0.897/0.900
Llama-3-8B | 0.602 | 0.930/0.904/0.908

Interface text:
AIE
aws

### Lesson 2

Vision-language model captions and user actions can serve as content annotations.

[5:02](https://www.youtube.com/watch?v=XdAWgO11zuk&t=302s) [VLM captions and user actions are useful annotations.](https://assets.ai.engineer/sites/aiecode2025/sha256/149377f4365792b134ccb29b34d6da7e762d053ee350a33f7557ed40dc5133fb.webp)

Lesson 2

VLM captions and user actions are useful annotations

AIE
Microsoft
smol ai

### Text Feature Performance

Adding text features sequentially raises accuracy from 0.457 to 0.580. User-curated board titles and high-engagement query tokens provide further gains as content annotations.

[6:19](https://www.youtube.com/watch?v=XdAWgO11zuk&t=379s) [A table shows accuracy rising from 0.457 to 0.580 as text features are added sequentially.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad95725cd9cc602306ea9788427f03dc771f7d7846eb12cb869b1e39bd8f69f8.webp)

Text Feature Performance

• user engagement can be an important source of annotation for contents
• enriched text features and metadata significantly contribute to building a more robust relevance model

Table 2: Benchmark the improvement with the sequential addition of text features.

Text Features | Accuracy | AUROC 3+/4+/5+
Synthetic image caption | 0.457 | 0.838/0.781/0.760
+ Pin title and description | 0.561 | 0.906/0.875/0.876
+ Link title and description | 0.565 | 0.910/0.880/0.880
+ User-curated board titles | 0.577 | 0.916/0.888/0.890
+ High-engagement query tokens | 0.580 | 0.917/0.892/0.895

Interface text:
AIE
Microsoft
smol ai

### Lesson 3: Knowledge distillation is the key to efficient serving

Pinterest uses knowledge distillation into smaller models to scale search relevance prediction while controlling serving costs.

[7:12](https://www.youtube.com/watch?v=XdAWgO11zuk&t=432s) [Lesson 3 states that knowledge distillation is the key to efficient serving.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7f7b60c3047cf234c6e34fef601a8725434e0ad6e613f59c4b280e6af1b5a15.webp)

Lesson 3
Knowledge distillation is the key to efficient serving

AIE
aws

[7:24](https://www.youtube.com/watch?v=XdAWgO11zuk&t=444s) [Lesson 3: Knowledge distillation is the key to efficient serving.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d899ca5addda6b1b4bd8ed3e2230fd650dd5bef910d55991f8bef433c70f078.webp)

Lesson 3
Knowledge distillation is the key to efficient serving

AIE
Microsoft
smol ai

### Distilling the relevance student model

Pin and query inputs feed embeddings and an MLP that predicts five-scale relevance. The student trains against the LLM teacher’s soft scores using cross-entropy loss.

[7:38](https://www.youtube.com/watch?v=XdAWgO11zuk&t=458s) [Pin and query inputs feed embedding layers and an MLP; student relevance scores are compared with LLM teacher predictions using cross-entropy loss.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9dd90e419ebe640c05d66fd0b649333d14c3f9d6704cb0701efc5a273d38d33.webp)

The production-served relevance student model is distilled with semi-supervised learning

5-scale relevance
LLM teacher model predictions
Cross entropy loss

MLP
Embedding layers + Feature crosses

Pin text encoder LM
Query encoder LM
Pin features
Query features
Pin-Query text-match statistics (eg. BM25)

### Scaling data with knowledge distillation

Human labels fine-tune an LLM teacher, which labels daily search-log samples. Those distilled labels train a relevance student served online, scaling the label set to about 100 times the human labels.

[8:00](https://www.youtube.com/watch?v=XdAWgO11zuk&t=480s) [Human labels fine-tune an LLM teacher, which labels search-log samples for training a relevance student model used in real-time serving.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac52b8cb01198bc20901e2f3dc5ea283d4eb6b83ce2f2ed5af34bc043fef1c2e.webp)

We use knowledge distillation to scale data O(100x), across domains, languages, and countries

human labels
fine-tune
LLM teacher model
offline inference
daily samples from search logs
distilled labels (~100× human labels)
relevance student model
relevance scores
real-time serving

### Language models in the production student

The production student embeds pins and queries separately, then combines their embeddings with other features and text-match statistics. An MLP predicts relevance on a five-point scale.

[9:10](https://www.youtube.com/watch?v=XdAWgO11zuk&t=550s) [Pin text and search queries pass through separate language models, alongside query, pin and text-match features, to produce relevance scores.](https://assets.ai.engineer/sites/aiecode2025/sha256/fde5aee02f5068b4c715688b623e54ce1b8bab97b24dcde0caf719371165f826.webp)

This production student model also has LLMs

5-scale relevance
MLP
Embedding layers + Feature crosses

Pin Emb
LM (xlm-roberta-base)
BLIP Image Caption
Title / Description
Rich Pin Text
Board Titles
Navboost Queries
Pin text Encoder LM

Query Emb
LM (xlm-roberta-base)
Search Query
Query Encoder LM

Query Features
Pin Features
Pin-Query text-match statistics (eg. BM25)

### Scaling bi-encoders with caching and offline inference

Pinterest computes pin embeddings offline for billions of pins and caches near-real-time query embeddings. The speaker reports around an 85% cache hit rate for queries.

[10:00](https://www.youtube.com/watch?v=XdAWgO11zuk&t=600s) [The student architecture highlights separately computed pin and query embeddings, with caching and inference annotations; part of the pin annotation is obscured.](https://assets.ai.engineer/sites/aiecode2025/sha256/9506cbb52f86878c25b669b9e4d0f02de8b918a769aa1b4e2ca6bd86c7c643c1.webp)

Bi-encoder LLMs can scale (caching + offline-inference go brrr)

5-scale relevance
MLP
Embedding layers + Feature crosses

Pin Emb
LM (xlm-roberta-base)
BLIP Image Caption
Title / Description
Rich Pin Text
Board Titles
Navboost Queries
Pin text Encoder LM
[illegible]ferred on billions of pins

Query Emb
LM (xlm-roberta-base)
Search Query
Query Encoder LM
Cached
near Real Time inferred

Query Features
Pin Features
Pin-Query text-match statistics (eg. BM25)

### Lesson 4

Relevance-tuned models also produce general-purpose embeddings for Pins and search queries that can be used across Pinterest.

[12:09](https://www.youtube.com/watch?v=XdAWgO11zuk&t=729s) [Lesson 4: Relevance-tuned LLMs produce rich general-purpose semantic representations.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c9eb4a8e0280c2ab13f205cbf46087ea74d4bbe7e3260afea3be50f6dd06684.webp)

Lesson 4

Relevance-tuned LLMs produce rich general-purpose semantic representations

### Production relevance student model

Pin and query encoders feed embeddings into feature crosses alongside other features. An MLP then predicts relevance on a five-point scale distilled from an LLM.

[12:21](https://www.youtube.com/watch?v=XdAWgO11zuk&t=741s) [Diagram of pin and query encoders, feature crosses, and an MLP producing five-scale relevance predictions.](https://assets.ai.engineer/sites/aiecode2025/sha256/98667fedd186d157cce9dfbaf208ab0aafde36b30cfef4932716a0ed8816bca7.webp)

Production relevance student model

5-scale relevance,
LLM distilled

MLP

Embedding layers + Feature crosses

Pin Emb
LM (xlm-roberta-base)
BLIP Image Caption
Title / Description
Rich Pin Text
Board Titles
Navboost Queries
Pin text
Encoder LM
[illegible]ferred on billions of pins

Query Emb
LM (xlm-roberta-base)
Search Query
Query
Encoder LM
Cached
near Real Time inferred

Query Features
Pin Features
Pin-Query text-match statistics (eg. BM25)

### LLM embeddings as powerful representations

The relevance student model produces pin and query embeddings that Pinterest can reuse to represent Pins, search queries, and boards.

[12:27](https://www.youtube.com/watch?v=XdAWgO11zuk&t=747s) [The student-model diagram emphasizes pin and query embeddings while fading the other model components.](https://assets.ai.engineer/sites/aiecode2025/sha256/6966aa3f127e9e03bbf666b6834fb9c48df6cb4dc95acfda05d45eb35d5970d3.webp)

LLM Embeddings from here are free, powerful representations

5-scale relevance,
LLM distilled

MLP

Embedding layers + Feature crosses

Pin Emb
LM (xlm-roberta-base)
BLIP Image Caption
Title / Description
Rich Pin Text
Board Titles
Navboost Queries
Pin text
Encoder LM
[illegible]ferred on billions of pins

Query Emb
LM (xlm-roberta-base)
Search Query
Query
Encoder LM
Cached
near Real Time inferred

Query Features
Pin Features
Pin-Query text-match statistics (eg. BM25)

### LLM Embedding Applications

Pinterest reports gains across embedding applications, including +16% Recall@10 for shopper retrieval and 2.7x improved precision on content moderation models.

[12:45](https://www.youtube.com/watch?v=XdAWgO11zuk&t=765s) [Six reported results for embedding applications, including gains in pin shares, repins, retrieval recall, moderation precision, and session duration.](https://assets.ai.engineer/sites/aiecode2025/sha256/204506df09c27d5bc3a4e2540c1fd326e5dcb9b81b4e72a1b4804a4e5a5058d3.webp)

LLM Embedding Applications

+1.1%
Pin Shares
Related Pins Ranking

+0.85%
Repins
Related Pins Fresh Retrieval

+16%
Recall@10 on
Shopper Embedding Retrieval

+1.4%
Repins on
Home Board Module Feed

2.7x
Improved precision on content moderation models

+0.12%
Sessions >5min from
Home Feed Module Ranking

### What We Learned from Using LLMs in Pinterest Search

Pinterest found LLMs useful for relevance prediction, VLM captions and user actions useful as annotations, and distillation effective for serving. Relevance tuning produced rich semantic representations.

[13:15](https://www.youtube.com/watch?v=XdAWgO11zuk&t=795s) [Four lessons cover relevance prediction, content annotations, knowledge distillation, and semantic representations from relevance tuning.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e513cec01f48ac5a332054760e1af57f5bfca59790ff6b5f1cce700ce42210d.webp)

What We Learned from Using LLMs in Pinterest Search

Lesson 1
LLMs are good at relevance prediction

Lesson 2
VLM captions and user actions are useful content annotations

Lesson 3
Knowledge distillation is the key to efficient serving

Lesson 4
Relevance-tuning produces rich semantic representations

### Thank you

The speakers close the talk and invite questions from the audience.

[14:10](https://www.youtube.com/watch?v=XdAWgO11zuk&t=850s) [Pinterest thank-you slide with a collage of images; contact text is partly obscured.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b5bcefb9fd4108980913459741c4cc77c4140b5b3b55804931ede57820c4c29.webp)

Thank you!

For any questions or feedback, please reach out to

[redacted]
[redacted]

## Full transcript

[0:00](https://www.youtube.com/watch?v=XdAWgO11zuk&t=0s) [on-hold electronic music] Yeah, hi, everyone.

[0:15](https://www.youtube.com/watch?v=XdAWgO11zuk&t=15s) Um, thanks for joining the talk today. Um, we are super excited to be here and to share some of the learnings we, um, we have from integrating the LLM into Pinterest Search.

[0:27](https://www.youtube.com/watch?v=XdAWgO11zuk&t=27s) My name's Han, and today I'll be presenting with Mukuntha, and we are both machine learning engineers from Search Relevance team at Pinterest.

[0:36](https://www.youtube.com/watch?v=XdAWgO11zuk&t=36s) So start with a brief introduction to Pinterest. Um, Pinterest is a visual discovery platform where Pinners can come to find inspiration to create a life they love. And there are three main discovery surfaces on Pinterest, the home feed, the related pins, and search.

[0:54](https://www.youtube.com/watch?v=XdAWgO11zuk&t=54s) In today's talk, we'll be focusing on search and, um, where the user can type in their queries and, um, find useful, inspiring content based on their information need. And we'll share, um, how we leverage LLM to improve the search relevance.

[1:13](https://www.youtube.com/watch?v=XdAWgO11zuk&t=73s) Um, here are some key statistics for Pinterest Search. Every month, we handle over six billion searches with billions of pins to search from, covering topics from recipe, home decor, travel, fashion, and beyond.

[1:29](https://www.youtube.com/watch?v=XdAWgO11zuk&t=89s) And at Pinterest, search is remarkably global and multilingual. We support over forty-five languages and are reaching Pinners in more than one hundred countries. These numbers highlight the importance of search at Pinterest and why we are investing, um, in search relevance to improving the search experience.

[1:52](https://www.youtube.com/watch?v=XdAWgO11zuk&t=112s) So, um, this is an overview of how Pinterest Search work at the back end. So it's similar to, um, many recommendation system at industry. It has query understanding, retrieval, reranking, and the blending stage, and finally produce, um, relevant and engagement search feeds.

[2:12](https://www.youtube.com/watch?v=XdAWgO11zuk&t=132s) And, um, in today's go- talk, we'll be focusing on the semantic relevance modeling that happen at the reranking stage and share about how we use LLM to improve, um, the search relevance on the search.

[2:29](https://www.youtube.com/watch?v=XdAWgO11zuk&t=149s) Okay, so, um, here's our search relevance model, which, um, is essentially a classification model. Given a search query and a pin, the model will predict how much the pin is relevant to the search query.

[2:44](https://www.youtube.com/watch?v=XdAWgO11zuk&t=164s) And to measure this, we use a five-point scale, um, ranging from the most relevant to most irrelevant.

[2:53](https://www.youtube.com/watch?v=XdAWgO11zuk&t=173s) All right. Um, now we are gonna share some key learnings we have from using the LLM to improve search, Pinterest search relevance. And here are four main takeaways that we would like to, um, go into more details.

[3:11](https://www.youtube.com/watch?v=XdAWgO11zuk&t=191s) Lesson one, LLMs are good at relevance prediction.

[3:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=197s) Um, so before I present, um, the result, let me first give a quick overview of the model architecture that we are using. Um, we concatenate the query and the pin text together and pass them into a LLM to get a, um, embedding.

[3:35](https://www.youtube.com/watch?v=XdAWgO11zuk&t=215s) So this is called, um, Cross-encoder structure, we- we-- where we can better capture the interaction between the query and the pin. And then we feed the, um, embedding from LLM into MLP layer to produce a five-dimensional vector which correspond to the, um, five relevance levels.

[3:55](https://www.youtube.com/watch?v=XdAWgO11zuk&t=235s) And during training, we fine-tune some open source LLM using Pinterest internal data and to better adapt the model to our Pinterest content.

[4:07](https://www.youtube.com/watch?v=XdAWgO11zuk&t=247s) And here, um, I'd like to share some results, um, to demonstrate that the usefulness of LLM. And as a baseline, we use SearchSage, which is a Pinterest in-house content and the query embedding.

[4:24](https://www.youtube.com/watch?v=XdAWgO11zuk&t=264s) And, um, so if you look at the table, you can see that the LLM has substantially, um, improved the performance of the relevance prediction. And as we use more advanced LLMs and increase the model size, the performance keeps improving.

[4:42](https://www.youtube.com/watch?v=XdAWgO11zuk&t=282s) And for example, um, the eight billion Llama three model gives, um, twelve percent of improvement over the multilingual BERT base model and twenty percent of improvement over the SearchSage embedding model.

[4:55](https://www.youtube.com/watch?v=XdAWgO11zuk&t=295s) So, um, the lesson here is that, um, LLMs, they are quite good at relevance prediction.

[5:04](https://www.youtube.com/watch?v=XdAWgO11zuk&t=304s) Um, lesson two, the vision language model generated captions and the user actions can be quite useful for content annotations.

[5:16](https://www.youtube.com/watch?v=XdAWgO11zuk&t=316s) So to use LLM for search, uh, for relevance prediction, we need to build a text representation of each pin. And here I listed several features that we used in our model.

[5:30](https://www.youtube.com/watch?v=XdAWgO11zuk&t=330s) Besides the, um, the title and description of the pin, we also include, um, the VLM-generated synthetic image caption to directly extract information from the image itself. And besides that, we add some, um, user engagement-based feature like the board titles, um, for the user-curated board that the pin has been saved to,

[5:55](https://www.youtube.com/watch?v=XdAWgO11zuk&t=355s) or, um, the queries that led to the highest engagement with this pin on search surface So these two user action-based features, um, serves as additional annotation for the content.

[6:09](https://www.youtube.com/watch?v=XdAWgO11zuk&t=369s) And, um, here the five source of feature together helps to build a more, um, robust and comprehensive text representation for each pin.

[6:22](https://www.youtube.com/watch?v=XdAWgO11zuk&t=382s) Uh, we... To understand the, um, importance of t-each raw text feature, we also did some ablation studies. We used the, um, VLM generated image caption as a baseline. And, um, as you can see, itself already pro-um, provide a very solid baseline.

[6:42](https://www.youtube.com/watch?v=XdAWgO11zuk&t=402s) And as we sequentially add more raw text feature, we keep seeing performance improvement. And this indicate that enriching the raw text feature is quite useful for relevance prediction. And notably, um, the last two rows of the table shows the performance gain we have by adding these user action-based features.

[7:04](https://www.youtube.com/watch?v=XdAWgO11zuk&t=424s) So these features turned out to be quite useful content annotation that help model better understand the content.

[7:14](https://www.youtube.com/watch?v=XdAWgO11zuk&t=434s) All right. Um, next, I will hand over to Mukuntha to talk about how we used knowledge distillation to productionize this model.

[7:22](https://www.youtube.com/watch?v=XdAWgO11zuk&t=442s) Right. Yeah. Uh, so now we have a good relevance model, which is good at predicting search relevance. Uh, but how do we actually scale this up without bankrupting Pinterest?

[7:33](https://www.youtube.com/watch?v=XdAWgO11zuk&t=453s) Uh, usually the answer is knowledge distillation into smaller models. Um, and this is the production served relevance student model that we distilled from the teacher model using semi-supervised learning.

[7:45](https://www.youtube.com/watch?v=XdAWgO11zuk&t=465s) Uh, the student model is trained to predict five-scale relevance scores too. Uh, it trains using the five, uh, scale soft scores pro-produced by the teacher model. Um, and we produce data for this using a semi-supervised learning setup that, uh, I'll show in the next slide.

[8:03](https://www.youtube.com/watch?v=XdAWgO11zuk&t=483s) So the LLM teacher model is trained on a small set of human labeled data that we get from human annotators who are trained in very specific segments. Uh, we fine-tune, and this is a multilingual language model which uses pretty generic features which scale across a lot of different domains, et cetera.

[8:22](https://www.youtube.com/watch?v=XdAWgO11zuk&t=502s) Um, and the way we get training data for the student, uh, is through, uh, sampling from daily search logs, which is, um, all the searches, uh, people make on Pinterest.

[8:34](https://www.youtube.com/watch?v=XdAWgO11zuk&t=514s) Uh, and since we sample daily, uh, this includes any trending queries, all the latest, freshest pins on Pinterest. And this is also remarkably go-global, like we mentioned, and only a small subset of this comes from the US, where most of our human labeled data comes from.

[8:52](https://www.youtube.com/watch?v=XdAWgO11zuk&t=532s) Um, we sample from this, and we label using the teacher, and we scale it up pretty much 100X, uh, across different domains, languages, countries where, uh, the LLM teacher model produces pretty good labels.

[9:05](https://www.youtube.com/watch?v=XdAWgO11zuk&t=545s) We train the student model, and this is the model that actually gets served online. Um,

[9:12](https://www.youtube.com/watch?v=XdAWgO11zuk&t=552s) and, uh, zooming into the student model, uh, also-- this also has language models in it. Uh, but unlike the teacher model, it's not a cross-encoder. It, uh, is a bi-encoder, uh, which essentially means we don't have cross interactions between the pin and the query, uh, representations.

[9:30](https://www.youtube.com/watch?v=XdAWgO11zuk&t=570s) Um, the pin gets embedded separately, query gets embedded separately, and it also uses a lot of other features like, um, SearchSage that we previously mentioned for both embedding the query and the pin.

[9:41](https://www.youtube.com/watch?v=XdAWgO11zuk&t=581s) Uh, we have GraphSAGE embeddings, which Pinterest has published papers on, um, and OmniSage and a lot of other embedding features for query and pin. But we also use, uh, a lot of pin query text match statistics like BM25, which we've seen historically perform really well for predicting search relevance.

[9:59](https://www.youtube.com/watch?v=XdAWgO11zuk&t=599s) Um, and the reason this scales well is the bi-encoder. Uh, bi-encoder large language models can scale really well, uh, when we, uh, use offline inference and caching. Uh, the pin embedding here is entirely offline inferred on billions of pins.

[10:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=617s) Uh, it uses predominantly the same text features that we mentioned on the teacher, uh, which helps distill efficiently. Um, and, uh, we only, uh, re-infer, uh, these embeddings every time that these inputs meaningfully change, uh, meaning that, uh, every time that we, uh, need new embeddings, um, it's only going to run on a few set of new

[10:40](https://www.youtube.com/watch?v=XdAWgO11zuk&t=640s) pins. Um, and, uh, this is offline inferred, so none of this is happening online when a user issues a search query. Uh, and the query embedding is pretty much, uh, real-time inferred online.

[10:52](https://www.youtube.com/watch?v=XdAWgO11zuk&t=652s) Uh, and search queries are pretty short. Um, they don't occupy too many tokens, which means, uh, we can keep the latencies for the query embedding up to like a few milliseconds.

[11:02](https://www.youtube.com/watch?v=XdAWgO11zuk&t=662s) Um, and we also cache this, uh, because search queries get repeated a lot, and we get around an eighty-five percent cache hit rate. Um, and yeah, this scales really well, uh, to actually serve Pinterest traffic.

[11:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=677s) Um, the online results here, uh, the first four numbers are relevance, uh, measurements, nDCG, uh, precision at eight, uh, measured on the US, Germany, France, uh, specific segments that we zoomed into.

[11:31](https://www.youtube.com/watch?v=XdAWgO11zuk&t=691s) Uh, we can actually see that we get relevance gains international, uh, internationally even though we started with a very limited set of US data for this particular experiment. Um, and, uh, we also see that search fulfillment, which measures engagement on search, um, fulfilling actions, um, also goes up, um, also on non-US even though our, uh, starting data

[11:56](https://www.youtube.com/watch?v=XdAWgO11zuk&t=716s) was predominantly US. And, uh, yeah, uh, large language models are very good at, uh, expanding across many different domains, countries, uh, even though, uh, they have-- weren't explicitly trained for this.

[12:10](https://www.youtube.com/watch?v=XdAWgO11zuk&t=730s) Um, and, uh, this is a bonus. Uh, we also found that relevance-tuned large language models produce really good rich, uh, semantic representations, which are very good general purpose. Uh, this is the same production relevance student model that I shared on the previous slide.

[12:27](https://www.youtube.com/watch?v=XdAWgO11zuk&t=747s) Uh, and, uh, the Pin embedding and the query embedding, uh, are basically free representations that we get from these models, uh, which can be used across Pinterest for representing Pins and search queries.

[12:41](https://www.youtube.com/watch?v=XdAWgO11zuk&t=761s) Um, we, we also use this to represent boards using the titles, et cetera. Um, and we found that using these embeddings, especially since they've been distilled from a large language model teacher and also have large, uh, language models in them, uh, they are very good at semantic content representations.

[13:01](https://www.youtube.com/watch?v=XdAWgO11zuk&t=781s) Uh, and yeah, they perform pretty well across, uh, related Pins, home feed, and a lot of other surfaces where we've seen, uh, representations improve by adding these things. Um, so let me go over the key takeaways again.

[13:18](https://www.youtube.com/watch?v=XdAWgO11zuk&t=798s) Um, I think lesson one, we found that LLMs are really good at relevance prediction. Uh, lesson two, we found that visual language model captions are good, uh, ex- good ways to imbue them with, uh, image representations, and, uh, user actions are very good content annotations.

[13:36](https://www.youtube.com/watch?v=XdAWgO11zuk&t=816s) Um, three, uh, we found that knowledge distillation is a very good way to scale, uh, and efficiently serve models, uh, online. And, uh, lesson four, uh, relevance tuning produces pretty rich representations, uh, that embed semantic representations for content fairly well.

[13:56](https://www.youtube.com/watch?v=XdAWgO11zuk&t=836s) Thank you. Um, I wonder if there are any questions from the audience. Please come up to the mics.

[14:03](https://www.youtube.com/watch?v=XdAWgO11zuk&t=843s) How did you decide which open source LLMs to fine-tune?

[14:09](https://www.youtube.com/watch?v=XdAWgO11zuk&t=849s) Yeah, that's a... Yeah, that's a very good question. So we did a lot of experiment trying different language models. And, um, in the previous slide, we also share some, um, performance for different language model.

[14:22](https://www.youtube.com/watch?v=XdAWgO11zuk&t=862s) Yeah. So we did a lot of experiment and find the one that gives us the best performance. Yeah.

[14:29](https://www.youtube.com/watch?v=XdAWgO11zuk&t=869s) Uh, if you could just walk us through somebody typing a search prompt. The confusion that I have is you have, like, LLMs, uh, building some sort of matching. Is it just being used for the label to be distilled, or how did you shim that into the bi-encoder?

[14:46](https://www.youtube.com/watch?v=XdAWgO11zuk&t=886s) It wasn't really clear on the two-tower, uh, offline and online and how the LLM search kind of influenced that.

[14:55](https://www.youtube.com/watch?v=XdAWgO11zuk&t=895s) We use LLMs to distill into a student model which predicts search relevance specifically and produces five scale relevance scores. Um, and it's served at the end of the search pipeline.

[15:06](https://www.youtube.com/watch?v=XdAWgO11zuk&t=906s) It's, uh, the re-ranking stage. Um, like every recommendation system, we have a lot of, uh, CGs, which are candidate generators. We have early stage ranking, and this is one of the things that sits further down the pipeline, which actually predicts search relevance scores and, uh, is used

[15:24](https://www.youtube.com/watch?v=XdAWgO11zuk&t=924s) right before blending to actually produce a feed. So I think it's very similar to most recommender systems.

[15:34](https://www.youtube.com/watch?v=XdAWgO11zuk&t=934s) So, uh, I have a... Excuse me. Uh, I have a question on how you evolved into this architecture. Like, I'm sure Pinterest has pre-LLM era search as well. Like, what limitations did you see in those systems that this new architecture solved for?

[15:56](https://www.youtube.com/watch?v=XdAWgO11zuk&t=956s) So, um, if I'm understanding correctly your question is about what's the difference between the new system with the-

[16:02](https://www.youtube.com/watch?v=XdAWgO11zuk&t=962s) Yeah

[16:02](https://www.youtube.com/watch?v=XdAWgO11zuk&t=962s) ... from the-

[16:03](https://www.youtube.com/watch?v=XdAWgO11zuk&t=963s) What, what was the, what was the driver to adapt, adapting LLMs, uh, for, for in, in your search pipeline? Did the type... Did it support new features, or is it...

[16:13](https://www.youtube.com/watch?v=XdAWgO11zuk&t=973s) Does it improve on the existing features where you had limitations?

[16:19](https://www.youtube.com/watch?v=XdAWgO11zuk&t=979s) I think they definitely improve, uh, especially with visual language model captions. I think we were es- very effectively able to expand beyond limited markets for actually measuring relevance data and getting relevance data.

[16:34](https://www.youtube.com/watch?v=XdAWgO11zuk&t=994s) And yeah, these multilingual models are very good at, uh, getting s- synthetic data for different markets for, uh, relevancy.

[16:47](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1007s) Hey. Um, g-great, great talk. Um, I was wondering why, uh, or if, if the embedding model is inherently multimodal, um, because you have text, which is, uh, the, the query, and then you're matching against, um, either text links or images.

[17:03](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1023s) And so how do you think about mo- multimodality?

[17:07](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1027s) It's definitely something we're exploring, but then, uh, on a lot of applications, I think we found, uh, visual captions are very good at ca- uh, capturing what the image has.

[17:17](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1037s) Um, and we have some very good captioning models in-house, which, uh, yeah, help us.

[17:24](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1044s) Great. Thanks.

[17:27](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1047s) What was the improvement-

[17:28](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1048s) Great talk. Uh, yeah, just a quick question. You mentioned that you saw improvements in other language as well. Did you start with the common baseline model for all languages, or did you have to sort of...

[17:40](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1060s) And just change the features for each language, or did you actually also start with separate l- models for individual languages?

[17:49](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1069s) Yeah, that's a-

[17:49](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1069s) Just curious how you actually saw the improvements manifest everywhere.

[17:54](https://www.youtube.com/watch?v=XdAWgO11zuk&t=1074s) Yeah. Um, so we s- we used the, um, same model for all languages, and we... Because we u- we are using the multilingual mo- um, LM, so we believe it can have transfer to other languages. [outro music]
