AI Engineer World's Fair 2024
AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale
Loading slides…
Read the talk
AI Frontiers in Trust and Safety: Combatting Multifaceted Harm on Tinder at Scale
Tinder’s approach combines carefully labeled platform data, small task-specific adapters, and shared-model inference to detect many kinds of harm under real-time constraints.
From a talk by Vibhor Kumar
Before you start: Familiarity with classification, fine-tuning, and precision versus recall will help; the article explains how adapters change training and serving.
Detecting harm across a wide spectrum
How do you build a detection system for violations that differ sharply in both frequency and severity? At Tinder, that problem sits within a broader trust and safety remit. Vibhor Kumar, who describes five years working on Tinder’s trust and safety systems, frames the work using four goals attributed to former Twitter trust and safety leader Del Harvey: prevent risk, reduce risk, detect harm, and mitigate harm. These goals protect users and the companies operating their platforms, but they still require ethical judgment. The engineering focus here is detecting harm.
The detection workload spans very different behaviors. Social media promotion in a profile is a relatively minor but prevalent violation of Tinder’s private-information policy, associated here with users who have low intent to participate on the dating platform. At the other end are lower-prevalence, higher-harm behaviors such as hate speech, harassment, and pig-butchering scams. A useful detector portfolio has to cover that spectrum: frequency alone cannot determine which categories deserve attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cheap generation changes the signals defenders can use
Generative AI first changes the volume and variety of material a platform must handle. Cheap production of misinformation, propaganda, and low-quality spam can drown out genuine content—a phenomenon Kumar calls content pollution. He also raises possible downstream copyright exposure for platforms hosting generated material. Accessible deepfake tools then lower the barrier to impersonation and catfishing, as well as malicious uses such as nonconsensual sexual imagery.
Organized fraud introduces a more specific detection problem. When attackers can generate fresh profile text and images for every account, hashes and similarity matching become less dependable. A campaign no longer has to reuse an obvious piece of content. That shifts attention toward signals tied to infrastructure and physical bottlenecks: IP addresses, ISP information, and phone numbers. Generated content may vary cheaply, while the infrastructure behind the accounts can still expose constraints. LLMs can also automate the messages those accounts send, extending automation from account creation into the interaction itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with semantic capability, then specialize
The same model ecosystem gives defenders a stronger starting point. Pretrained open models already contain substantial semantic knowledge and often support many languages. Few-shot prompting of Llama 3 or Mistral can provide an initial violation detector without training a language model from scratch. Kumar reports that fine-tuned open models can outperform few-shot GPT-4 on some downstream textual detection tasks, though he supplies no task-level evaluation details for that comparison.
Mature libraries now cover the path from model development through production serving, reducing the amount of low-level training machinery a team must build. Kumar reports that fine-tuning pretrained models with this tooling can shorten model development from months to weeks. The benefit is especially relevant when the target behavior keeps changing.
In adversarial detection, deploying a successful rule changes the environment: block a spam wave, and attackers have an incentive to alter their behavior until they can get through again. Kumar attributes slower model degradation to LLMs’ ability to generalize beyond the exact patterns seen during training. This does not end the adaptation cycle, but it can extend the useful life of a detector between updates.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the training contract small and explicit
The first major implementation step is assembling the training dataset, which Kumar identifies as one of the hardest parts. For some fine-tuning tasks, hundreds to thousands of examples can be sufficient. That is not a universal sample-size requirement; it makes the quality of each example particularly consequential.
For dataset design, treat a generative model—including GPT-4, Claude Opus, and open Llama or Mistral models—as a text-in, text-out function. The input is potentially violating text wrapped in a prompt. The target output is either a classification label or the characters that should be extracted as evidence of the violation. The model does not need to write an explanation when the product needs a classification.
Kumar illustrates this contract with a synthetic written bio that discloses an underage user. An illustrative record for the same task makes the distinction between the submitted text and its supervised target explicit:
json
{
"input": "Does this bio explicitly state that the user is younger than 18? Return YES or NO.\nBio: I am 17 and enjoy hiking.",
"output": "YES"
}
Here, the prompt defines a narrow decision and output contains the training label. An extraction task would instead put the relevant characters in that output field. Both use the same basic text-to-text training contract.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mine real platform examples, then verify the labels
At this dataset size, manual assembly is feasible. Larger LLMs offer another route, but the source of their inputs matters:
| Method | Main advantage | Main limitation |
|---|---|---|
| Manual assembly | Direct human judgment | Human labeling effort |
| Few-shot synthetic generation | Produces candidate examples readily | May miss the platform’s real distribution |
| LLM labeling of platform data | Starts from real platform content | Predictions still require verification |
Purely synthetic examples can be platform-agnostic even when they look plausible. Tinder found a hybrid approach more useful: use GPT-4 to make predictions on internal analytics data and mine training examples from those predictions. Kumar mentions prompting to handle alignment-related refusals, but gives no prompt recipe.
The economics depend on prevalence. To collect a fixed number of harmful examples, a rarer violation requires screening more material; Kumar describes the resulting cost in Tinder’s use as negligible. Estimating prevalence is also useful to trust and safety operations, beyond the immediate task of building a dataset. Heuristics can reduce model calls by selecting more likely candidates before the large model sees them.
The final step is human verification. Once the large model has mined candidate examples, reviewers correct mislabeled records and make policy judgments where a label is ambiguous. The large model helps find and provisionally label material; operations and policy expertise still determine the training targets.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why the production detector uses owned weights
If GPT-4 can help label the data, why not call it directly for every production decision? For Tinder’s real-time volume of profile interactions, Kumar identifies cost, latency, and throughput as the limiting factors. A large hosted model can be useful during dataset construction without being the right serving architecture for every interaction.
Control over the model also changes maintenance. With owned weights, the team can fine-tune again when detection quality degrades while keeping the underlying base model fixed. For classification, access to the output probability distribution additionally allows the team to derive a prediction-confidence score. That supplies a useful decision signal, although access to probabilities alone does not establish that the scores are calibrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train adapters instead of a separate full model for every task
The training ecosystem offers several levels of control:
- Code and notebooks: Hugging Face libraries support training pipelines in relatively little code; Kumar reports success building these pipelines in notebooks.
- Configuration-driven training: Axolotl, Ludwig, and LLaMA-Factory move much of the setup into configuration files.
- Interfaces for experimentation: H2O LLM Studio provides a framework with a no-code GUI, while Predibase offers managed tooling. Kumar highlights UI and dataset-management support and reports particular success with Predibase.
These choices reduce the mechanics of setting up experiments, leaving dataset quality and task definition as central engineering work.
Parameter-efficient fine-tuning is the next important reduction. LoRA, or low-rank adaptation, freezes the base model’s weights and learns a small set of additional adapter weights. Kumar describes adapters of a few megabytes in this workflow, rather than a separate full copy of every model’s weights. Training only that small addition makes rapid experiments on one or a few GPUs practical and opens up the use of larger base models.
Kumar reports that parameter-efficient adaptation of a larger base model is often more effective than fully fine-tuning a smaller model, particularly for classification. He also reports success with QLoRA, which combines quantization with adapter training to reduce memory requirements further. Its single-GPU feasibility depends on model size, hardware, and training configuration; it is not a promise that any model will fit. The adapter structure has another consequence beyond training efficiency: many specialized detectors can share a base model during inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Limit expensive inference and keep outputs short
Kumar reports tens of queries per second at roughly 100 milliseconds of latency for seven-billion-parameter models on A10 GPUs. This is Tinder’s reported operating result, not a general benchmark: input lengths, batching, GPU count, and the latency percentile are unspecified. It is sufficient for some real-time use cases, while higher-frequency domains need additional request filtering.
One such filter is deliberately simple. For social media detection, the system can restrict predictions to bios containing a word that is absent from a dictionary. The heuristic selects candidates for the model; it does not establish that a violation occurred. Its purpose is to reduce the number of requests reaching the more expensive detector.
Tinder is also exploring a classification cascade through distillation. A smaller-base adapter is trained for high recall, and the larger-base adapter is called only when the smaller model’s score exceeds a threshold. The first stage therefore screens broadly before committing larger-model compute. This is described as exploratory work, distinct from the deployed LoRAX serving system.
Finally, the output contract itself reduces inference work. Autoregressive generation produces tokens one at a time, so a long explanation adds repeated decoding steps. A classification or extraction task can need only one or a few output tokens. Returning a compact label or a short extracted span keeps the generation portion of prediction small.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantic understanding helps with both accuracy and evasion
Compared with earlier NLP models, Kumar reports substantial improvements in precision and recall, attributing them to the semantic capability of modern LLMs. Kumar reports near-100% recall on simpler tasks such as social-handle detection. The talk does not specify the evaluation set or precision operating point, and it describes improvements on more semantically complex tasks without quantifying them.
The practical test extends beyond performance on familiar examples. Users trying to evade detection introduce intentional typos, mix letters with numbers, and use innuendo. These changes can defeat a detector tied closely to particular strings while leaving the underlying meaning recognizable. Kumar reports that LLM-based adapters interpret such variations better and consequently become stale less quickly than traditional machine-learning detectors. That resilience matters because the system must continue working after users discover what it catches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend the process across modalities and the long tail
The next area of exploration is non-textual content. Kumar points to pretrained vision-language models such as LLaVA as a possible foundation for explicit-image detection. This is an active investigation, not a claim that Tinder has already deployed that moderation capability.
The broader goal is to make new harm detectors routine to build. Trust and safety operations and policy experts would create high-quality datasets with AI assistance; automated pipelines would train and retrain adapters; LoRAX would make each new adapter inexpensive to add to inference. The deployed text-serving architecture provides the foundation, while broader automation and coverage of the long tail remain the roadmap. A defense built this way can keep incorporating new categories and changing behavior, with a safer, healthier platform as its objective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Inference server for sharing a base model across many dynamically loaded LoRA adapters, with setup and client examples.
Original paper explaining how low-rank trainable matrices adapt frozen pretrained models.
Combines four-bit base-model quantization with LoRA and reports fine-tuning a 65B model on a single 48GB GPU.
Project repository for the fine-tuning tool named in the talk.
Low-code framework for building custom language models and other machine-learning models.
Unified framework for efficient fine-tuning of language and vision-language models.
Open-source fine-tuning framework with a no-code GUI and tools for comparing experiments.
Visual instruction-tuning project with architecture explanations and links to models, code, and data.
Full transcript
Loading transcript…