AI Engineer World's Fair 2024

AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale

Vibhor Kumar· AI Engineer, Tinder14:36

Loading slides…

Read the talk

AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale

Tinder’s approach combines carefully labeled platform data, small task-specific adapters, and shared-model inference to detect many kinds of harm under real-time constraints.

From a talk by Vibhor Kumar

Before you start: Familiarity with classification, fine-tuning, and precision versus recall will help; the article explains how adapters change training and serving.

Detecting harm across a wide spectrum

How do you build a detection system for violations that differ sharply in both frequency and severity? At Tinder, that problem sits within a broader trust and safety remit. Vibhor Kumar, who describes five years working on Tinder’s trust and safety systems, frames the work using four goals attributed to former Twitter trust and safety leader Del Harvey: prevent risk, reduce risk, detect harm, and mitigate harm. These goals protect users and the companies operating their platforms, but they still require ethical judgment. The engineering focus here is detecting harm.

The detection workload spans very different behaviors. Social media promotion in a profile is a relatively minor but prevalent violation of Tinder’s private-information policy, associated here with users who have low intent to participate on the dating platform. At the other end are lower-prevalence, higher-harm behaviors such as hate speech, harassment, and pig-butchering scams. A useful detector portfolio has to cover that spectrum: frequency alone cannot determine which categories deserve attention.

Slide showing a prevalence-to-harm arrow above four categories: social media promotion, underage users, harassment and hate speech, and pig butchering scams. The speaker inset partly covers the first example.Open full source frame
Tinder’s trust and safety examples span high-prevalence, low-harm behavior to low-prevalence, high-harm behavior.
0:161:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Cheap generation changes the signals defenders can use

Generative AI first changes the volume and variety of material a platform must handle. Cheap production of misinformation, propaganda, and low-quality spam can drown out genuine content—a phenomenon Kumar calls content pollution. He also raises possible downstream copyright exposure for platforms hosting generated material. Accessible deepfake tools then lower the barrier to impersonation and catfishing, as well as malicious uses such as nonconsensual sexual imagery.

Organized fraud introduces a more specific detection problem. When attackers can generate fresh profile text and images for every account, hashes and similarity matching become less dependable. A campaign no longer has to reuse an obvious piece of content. That shifts attention toward signals tied to infrastructure and physical bottlenecks: IP addresses, ISP information, and phone numbers. Generated content may vary cheaply, while the infrastructure behind the accounts can still expose constraints. LLMs can also automate the messages those accounts send, extending automation from account creation into the interaction itself.

2:252:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:25 · section reference included

Start with semantic capability, then specialize

The same model ecosystem gives defenders a stronger starting point. Pretrained open models already contain substantial semantic knowledge and often support many languages. Few-shot prompting of Llama 3 or Mistral can provide an initial violation detector without training a language model from scratch. Kumar reports that fine-tuned open models can outperform few-shot GPT-4 on some downstream textual detection tasks, though he supplies no task-level evaluation details for that comparison.

Mature libraries now cover the path from model development through production serving, reducing the amount of low-level training machinery a team must build. Kumar reports that fine-tuning pretrained models with this tooling can shorten model development from months to weeks. The benefit is especially relevant when the target behavior keeps changing.

In adversarial detection, deploying a successful rule changes the environment: block a spam wave, and attackers have an incentive to alter their behavior until they can get through again. Kumar attributes slower model degradation to LLMs’ ability to generalize beyond the exact patterns seen during training. This does not end the adaptation cycle, but it can extend the useful life of a detector between updates.

A green-bordered opportunities panel lists pretrained LLMs, fine-tuning, open-source tools, faster model development, and generalization to adversarial trends.Open full source frame
Generative AI opportunities for engineers and trust and safety organizations.
3:544:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

Make the training contract small and explicit

The first major implementation step is assembling the training dataset, which Kumar identifies as one of the hardest parts. For some fine-tuning tasks, hundreds to thousands of examples can be sufficient. That is not a universal sample-size requirement; it makes the quality of each example particularly consequential.

For dataset design, treat a generative model—including GPT-4, Claude Opus, and open Llama or Mistral models—as a text-in, text-out function. The input is potentially violating text wrapped in a prompt. The target output is either a classification label or the characters that should be extracted as evidence of the violation. The model does not need to write an explanation when the product needs a classification.

Kumar illustrates this contract with a synthetic written bio that discloses an underage user. An illustrative record for the same task makes the distinction between the submitted text and its supervised target explicit:

json

{
  "input": "Does this bio explicitly state that the user is younger than 18? Return YES or NO.\nBio: I am 17 and enjoy hiking.",
  "output": "YES"
}

Here, the prompt defines a narrow decision and output contains the training label. An extraction task would instead put the relevant characters in that output field. Both use the same basic text-to-text training contract.

5:375:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:37 · section reference included

Mine real platform examples, then verify the labels

At this dataset size, manual assembly is feasible. Larger LLMs offer another route, but the source of their inputs matters:

MethodMain advantageMain limitation
Manual assemblyDirect human judgmentHuman labeling effort
Few-shot synthetic generationProduces candidate examples readilyMay miss the platform’s real distribution
LLM labeling of platform dataStarts from real platform contentPredictions still require verification

Purely synthetic examples can be platform-agnostic even when they look plausible. Tinder found a hybrid approach more useful: use GPT-4 to make predictions on internal analytics data and mine training examples from those predictions. Kumar mentions prompting to handle alignment-related refusals, but gives no prompt recipe.

The economics depend on prevalence. To collect a fixed number of harmful examples, a rarer violation requires screening more material; Kumar describes the resulting cost in Tinder’s use as negligible. Estimating prevalence is also useful to trust and safety operations, beyond the immediate task of building a dataset. Heuristics can reduce model calls by selecting more likely candidates before the large model sees them.

The final step is human verification. Once the large model has mined candidate examples, reviewers correct mislabeled records and make policy judgments where a label is ambiguous. The large model helps find and provisionally label material; operations and policy expertise still determine the training targets.

6:517:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:51 · section reference included

Why the production detector uses owned weights

If GPT-4 can help label the data, why not call it directly for every production decision? For Tinder’s real-time volume of profile interactions, Kumar identifies cost, latency, and throughput as the limiting factors. A large hosted model can be useful during dataset construction without being the right serving architecture for every interaction.

Control over the model also changes maintenance. With owned weights, the team can fine-tune again when detection quality degrades while keeping the underlying base model fixed. For classification, access to the output probability distribution additionally allows the team to derive a prediction-confidence score. That supplies a useful decision signal, although access to probabilities alone does not establish that the scores are calibrated.

8:038:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:03 · section reference included

Train adapters instead of a separate full model for every task

The training ecosystem offers several levels of control:

  • Code and notebooks: Hugging Face libraries support training pipelines in relatively little code; Kumar reports success building these pipelines in notebooks.
  • Configuration-driven training: Axolotl, Ludwig, and LLaMA-Factory move much of the setup into configuration files.
  • Interfaces for experimentation: H2O LLM Studio provides a framework with a no-code GUI, while Predibase offers managed tooling. Kumar highlights UI and dataset-management support and reports particular success with Predibase.

These choices reduce the mechanics of setting up experiments, leaving dataset quality and task definition as central engineering work.

Parameter-efficient fine-tuning is the next important reduction. LoRA, or low-rank adaptation, freezes the base model’s weights and learns a small set of additional adapter weights. Kumar describes adapters of a few megabytes in this workflow, rather than a separate full copy of every model’s weights. Training only that small addition makes rapid experiments on one or a few GPUs practical and opens up the use of larger base models.

Kumar reports that parameter-efficient adaptation of a larger base model is often more effective than fully fine-tuning a smaller model, particularly for classification. He also reports success with QLoRA, which combines quantization with adapter training to reduce memory requirements further. Its single-GPU feasibility depends on model size, hardware, and training configuration; it is not a promise that any model will fit. The adapter structure has another consequence beyond training efficiency: many specialized detectors can share a base model during inference.

8:539:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:53 · section reference included

Serve a portfolio of detectors through one shared base

Tinder uses LoRAX in production to serve these adapters. The architecture shares the expensive base model across specialized fine-tunes, then shuffles adapters and batches requests for efficient joint execution. Kumar describes LoRAX as supporting thousands of fine-tuned adapters on a single GPU. That is a claim about serving a portfolio of adapters, not processing unlimited simultaneous requests.

Sharing the base changes the incremental cost of adding a category. Kumar characterizes the marginal cost of another adapter on the same base as virtually zero. The useful implication is that hate speech, promotion, catfishing, and pig-butchering detection can have distinct adapters without each requiring a dedicated full-model deployment. Request volume still consumes compute, but adding a detector need not mean adding another dedicated GPU service.

The deployment operation is correspondingly small:

  1. Store the trained adapter weights on a filesystem accessible to the serving system.
  2. Modify the LoRAX client request to select that adapter.

The detector becomes another specialization of the shared base. Kumar credits Predibase with developing and maintaining LoRAX and supporting Tinder’s deployment.

10:2110:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:21 · section reference included

Limit expensive inference and keep outputs short

Kumar reports tens of queries per second at roughly 100 milliseconds of latency for seven-billion-parameter models on A10 GPUs. This is Tinder’s reported operating result, not a general benchmark: input lengths, batching, GPU count, and the latency percentile are unspecified. It is sufficient for some real-time use cases, while higher-frequency domains need additional request filtering.

One such filter is deliberately simple. For social media detection, the system can restrict predictions to bios containing a word that is absent from a dictionary. The heuristic selects candidates for the model; it does not establish that a violation occurred. Its purpose is to reduce the number of requests reaching the more expensive detector.

Production slide describing LoRAX, batching small LoRA adapters, and virtually zero marginal cost per new adapter, beside a cartoon mascot and project banner. The speaker inset obscures some lower text.Open full source frame
LoRAX serving uses adapter shuffling and batching to support real-time inference.

Tinder is also exploring a classification cascade through distillation. A smaller-base adapter is trained for high recall, and the larger-base adapter is called only when the smaller model’s score exceeds a threshold. The first stage therefore screens broadly before committing larger-model compute. This is described as exploratory work, distinct from the deployed LoRAX serving system.

Finally, the output contract itself reduces inference work. Autoregressive generation produces tokens one at a time, so a long explanation adds repeated decoding steps. A classification or extraction task can need only one or a few output tokens. Returning a compact label or a short extracted span keeps the generation portion of prediction small.

11:2311:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:23 · section reference included

Semantic understanding helps with both accuracy and evasion

Compared with earlier NLP models, Kumar reports substantial improvements in precision and recall, attributing them to the semantic capability of modern LLMs. Kumar reports near-100% recall on simpler tasks such as social-handle detection. The talk does not specify the evaluation set or precision operating point, and it describes improvements on more semantically complex tasks without quantifying them.

The practical test extends beyond performance on familiar examples. Users trying to evade detection introduce intentional typos, mix letters with numbers, and use innuendo. These changes can defeat a detector tied closely to particular strings while leaving the underlying meaning recognizable. Kumar reports that LLM-based adapters interpret such variations better and consequently become stale less quickly than traditional machine-learning detectors. That resilience matters because the system must continue working after users discover what it catches.

12:3412:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:34 · section reference included

Extend the process across modalities and the long tail

The next area of exploration is non-textual content. Kumar points to pretrained vision-language models such as LLaVA as a possible foundation for explicit-image detection. This is an active investigation, not a claim that Tinder has already deployed that moderation capability.

The broader goal is to make new harm detectors routine to build. Trust and safety operations and policy experts would create high-quality datasets with AI assistance; automated pipelines would train and retrain adapters; LoRAX would make each new adapter inexpensive to add to inference. The deployed text-serving architecture provides the foundation, while broader automation and coverage of the long tail remain the roadmap. A defense built this way can keep incorporating new categories and changing behavior, with a safer, healthier platform as its objective.

Pink slide titled The Future branches into non-textual modalities, including image detection with fine-tuned VLMs, and rapidly developing more adapters across the long tail of harm.Open full source frame
Future directions: non-textual modalities and more adapters for long-tail harm detection.
13:2613:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:26 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…