AI Engineer World's Fair 2024

Iterating on LLM apps at scale: Learnings from Discord

Ian Webster· Senior Staff Software Engineer, Discord18:26

Loading slides…

Read the talk

Iterating on LLM Apps at Discord Scale

Discord’s experience with Clyde shows how small evaluations, familiar development tools, and deliberate adversarial testing can make LLM risks easier to measure before launch.

From a talk by Ian Webster

Before you start: Familiarity with system prompts, tool calls, retrieval-augmented generation, and basic unit testing will help you follow the examples.

When rare failures become a launch problem

How do you ship a conversational assistant without having it teach children how to build bombs? For Discord’s Clyde AI, that was a harder problem than choosing models or fine-tuning them. Clyde combined chat, agent behavior, and retrieval-augmented generation. Ian Webster describes its launch as reaching over 200 million Discord users—a characterization of rollout scale, not a measured count of active Clyde users.

The recurring launch blockers were security, legal, safety, and sometimes policy. Dangerous instructions were only one concern; harassment and racism created other ways for the product to cause harm. Getting those stakeholders comfortable required a way to quantify risk before users discovered the failures in production.

Slide titled “What's stopping you from shipping?” with three bullets about launch blockers, quantifying risk, and failures discovered in production.Open full source frame
Security, legal, and safety concerns can block LLM launches.

Webster illustrates the scale problem with a hypothetical: a one-in-a-million failure rate across 200 million opportunities implies 200 expected failures. That is an expected count, not an observed incident total, and users are not interchangeable with model responses. Rare failures still require measurement and mitigation when a product operates at enormous scale.

1:061:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:06 · section reference included

Treat evaluations as small unit tests

An evaluation systematically characterizes a system by supplying inputs and measuring outputs. At Discord, that meant testing both sides of the product requirement: create a good experience and reduce the risk of harm. The useful starting point was ordinary unit testing, rather than elaborate scoring machinery.

Break the application into the decisions and operations it performs. Moderation gets its own evaluation; each tool usage gets another. An end-to-end evaluation then supplements those focused suites. Most checks should stay small, fast, and ideally deterministic, so a failure points toward a specific part of the system rather than merely saying that the whole application produced a bad answer.

2:503:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:50 · section reference included

A lowercase letter as a personality check

Clyde was supposed to sound casual. That requirement could invite an LLM grader, a trained classifier, and rounds of temperature or hyperparameter tuning. Discord instead found a much cheaper signal: does the output begin with a lowercase letter? The check does not capture every aspect of personality, but it directly tests a visible feature of the desired conversational style.

A JavaScript implementation makes the narrow contract explicit:

javascript

function beginsWithLowercase(output) {
  return /^[a-z]/.test(output);
}

const examples = [
  "hey, what's up?",
  "Hey, what's up?",
  ""
];

for (const output of examples) {
  console.log({ output, passes: beginsWithLowercase(output) });
}

The first response passes; the capitalized response and empty string fail. This checks the first character rather than rewriting the response or asking another model to judge it. Webster informally estimates that the lowercase proxy delivered more than 80 percent of the benefit for about 1 percent of the work. Those figures describe his assessment of this particular shortcut, not measured classification accuracy.

Bell-curve meme contrasting lowercase checks at both ends with model grading, hyperparameters, classifiers, and other evaluation methods in the middle.Open full source frame
A simple lowercase check as a metric for casual chat personality.

The resulting personality mattered as a product feature. Webster follows the metric with humorous Clyde interactions that draw laughter from the audience. He defends those exchanges as time well spent: making users happy was part of the purpose, even if critics did not consider the interactions useful.

4:134:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:13 · section reference included

Separate the search decision from summarization

The same testing philosophy applies to web search followed by generation. There are two different questions: whether the agent should search, and whether it can use the retrieved material correctly. Combining them in every test makes those behaviors harder to distinguish.

SuiteInput under testBehavior checked
Tool triggeringA request that may need searchWhether the agent invokes the tool
SummarizationFixed webpage contextWhether the model summarizes it correctly

The summarization suite receives static context. It does not connect to a live database or run a live web search. Keeping the source material fixed lets the test focus on generation instead of changes in the retrieved content.

5:145:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:14 · section reference included

More prompt instructions can make results worse

As usage grew, the trade-off between model cost and accuracy became harder to manage. Prompt complexity introduced a second problem. Every evaluation failure tempted the team to add another exception to the prompt. Eventually those additions produced diminishing returns, then negative returns. Removing instructions and leaving the model room to choose a reasonable response improved the result.

Prompts can also create vendor lock-in. A prompt refined against GPT is not automatically a fair test of Claude or Llama. Webster attributes part of OpenAI’s advantage to developers becoming accustomed to GPT-style prompting. When evaluating another model family, spend time adapting its prompt rather than treating the existing prompt as a neutral specification.

5:586:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:58 · section reference included

Make evaluations an everyday development reflex

Treating evaluations as tests has consequences for the development environment. Developers should be able to run them locally without a mandatory cloud evaluation service. Webster’s airplane example includes an essential condition: the model must also run locally. Local orchestration alone does not make remote inference available offline.

The metrics should be as understandable as ordinary unit-test assertions. Complex logic inside the test makes it harder to know what a passing result means. Basic deterministic checks helped Discord make evaluation routine; Webster’s workflow target was for developers to run evaluations dozens of times a day as a quick command-line reflex.

Every pull request received an evaluation, Webster reports. The initial review process could be simple:

  1. Run the evaluation for the change.
  2. Paste a link to its results into the pull request.
  3. Integrate the checks into CI/CD as the workflow develops.

The important adoption step was making evaluation part of normal review, without requiring a separate elaborate product or process.

Slide listing local execution, independence from cloud services, understandable metrics, frequent developer runs, and an eval for every PR.Open full source frame
Building an eval culture: treat evaluations as tests.

Promptfoo, the open-source project Webster helps maintain, provided a CLI with local execution and declarative configuration. Its role in this workflow was practical: make tests easy for developers to define, run, and inspect.

7:247:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:24 · section reference included

Monitor production with the tools already in use

Discord used Datadog for LLM observability. Webster jokingly introduces it as a stealth AI startup, but the point is its familiarity: the team could put the LLM metrics it cared about alongside the rest of its product data. An existing observability system was sufficient for that job.

The team also implemented online production evaluations, including some simple one-shot checks graded by models. Those results flowed into Datadog too. The preference for deterministic development checks did not exclude model grading where the team found it useful in production.

Monitoring did not automatically produce a reusable evaluation dataset. The ideal feedback loop would incorporate live interactions into future tests, but Webster says privacy and other constraints prevented Discord from closing that loop. Instead, the team collected examples through internal dogfooding and public reports on Twitter and Reddit. Both failures and successful interactions became useful evaluation material.

9:229:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:22 · section reference included

Version prompts and experiment with routing

Prompt management stayed straightforward. Git was the source of truth for versioning, while a basic Retool application let nontechnical colleagues toggle configuration. Webster presents this as a workable arrangement for the team, while acknowledging that better alternatives might exist.

Routing addressed a more unusual problem. During long conversations, lower-powered models—including Llama and GPT-3.5—could drift away from their system prompts. Clyde gradually reverted toward a generic ChatGPT personality, which users disliked.

Discord tried randomly inserting an occasional GPT-4 response into the conversation. Webster likens it to a bowling alley bumper: a stronger response could steer the conversation back toward the intended personality. He says the experiment worked okay, but questions whether it was a smart general approach. The account provides neither a controlled comparison nor a recommended insertion frequency.

11:1311:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:13 · section reference included

A persona can become an attack surface

An internet-facing conversational product also attracts users who deliberately try to break it. Webster jokes about Discord’s audience to make that operational point: adversarial use was central to the problem, not an obscure edge case.

The Grandma Jailbreak made the risk concrete. Webster says it originated on Discord and describes himself as its first victim. The request wrapped dangerous content in a sentimental role-playing scenario: a deceased grandmother telling bedtime stories. Clyde adopted the grandmother persona and supplied harmful instructions. The failure was the model allowing the narrative frame to override the safety boundary. Public reaction and technology-media coverage increased the pressure to address it.

12:4112:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:41 · section reference included

Assess the application before deployment

Safeguarding an LLM application involves two complementary kinds of work:

StageWork
Before deploymentRisk assessment, red teaming, compliance, and legal review
During operationLive filtering

Webster saw more available solutions for filtering, but considered advance assessment the more important side. The team built a risk-assessment view in Promptfoo that grouped findings into categories such as brand and legal risk. Organizations could focus on the categories relevant to their own applications.

The baseline tests used an unaligned model to generate harmful inputs. Examples included requests about evading detection while stealing and requests involving child sexual abuse. These were synthetic test cases, not a collection of people manually writing every toxic example. Webster’s warning is that public applications will encounter such requests quickly; at the time of the talk, he found that most state-of-the-art models refused the direct requests outright.

13:5214:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:52 · section reference included

Test beyond direct harmful requests

A base model’s refusal behavior does not establish the safety of the application built around it. Adding system prompts and application behavior creates new attack vectors. To probe those interactions, Webster describes an attacker-and-judge loop:

  1. An attacker LLM proposes an adversarial input for the target application.
  2. The target produces a response.
  3. A judge guides the attacker’s revision, including changes in wording, euphemisms, or synonyms.
  4. The revised input is tested again.

This makes the attack responsive to the application’s behavior instead of stopping after one direct request.

Webster reports that the iterative approach elicited more harmful responses than the direct-input examples. One demonstration reframed a child-abuse request around a fictional antagonist in a crime novel. He tentatively identifies the models involved as Mistral and GPT, without establishing exact versions or their roles. A further example targeted school-violence planning. The substantive finding is that these revisions exposed weaknesses in safeguards that direct requests had not revealed.

15:5116:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:51 · section reference included

Safety testing extends beyond refusals

The final expansion is from harmful-content testing to the broader behavior of the application:

  • PII leakage: exposing personally identifiable information.
  • Hallucination: producing unsupported or fabricated information.
  • Task hijacking: diverting the application from its intended function, such as getting a specialized assistant to do unrelated homework.
  • Political opinions: producing political positions that an organization may consider inappropriate for its application.

These categories require more than a single check for whether the model refuses an obviously harmful request. They ask whether the application preserves its purpose and constraints across different kinds of input. Webster closes by offering the completely open-source Promptfoo project for both ordinary evaluation and red teaming.

17:3117:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:31 · section reference included

Resources

From the talk

  • Open-source tooling for evaluating prompts, agents, and RAG applications and running adversarial tests.

Updates since the talk

Transcript download
Full transcript

Loading transcript…