AI Engineer World's Fair 2025

Benchmarks Are Memes: How What We Measure Shapes AI—and Us

Alex Duffy15:44

Loading slides…

Read the talk

Benchmarks Are Memes: How What We Measure Shapes AI—and Us

A benchmark can turn a personal curiosity into a development target. AI Diplomacy shows what becomes visible when the test includes negotiation, deception and human judgment.

From a talk by Alex Duffy

How a personal test becomes a public target

What makes a test memorable enough that other people start using it? Alex Duffy opens with a meme generated by Claude Opus, then turns to the older meaning of the word. Leading AI training and consulting at Every, he approaches benchmarks as a form of education: they teach people what to notice about AI, while giving developers something to improve.

Richard Dawkins coined meme in the 1970s to describe ideas that spread. Christianity, democracy and capitalism are Duffy’s examples of that broader category. A benchmark can travel in the same way: someone proposes a recognizable challenge, other people repeat it, and the challenge becomes part of how a community understands capability.

Slide titled “Ideas That Spread” shows a cross, scales and a dollar sign, labeled Christianity, Democracy and Capitalism, with a Richard Dawkins attribution below.Open full source frame
Memes as ideas that spread: Christianity, democracy and capitalism.

Simon Willison’s pelican riding a bicycle makes this concrete. Duffy describes a personal test reaching Google I/O’s keynote roughly a year after Willison began using it. Counting the Rs in strawberries followed a similar cultural path: an easily understood failure became an iconic test, and Duffy says models had stopped making that mistake by the time of the talk. Sometimes even the name carries the idea. Humanity’s Last Exam attracted attention beyond AI circles because its title made the stakes immediately legible.

0:000:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From an interesting failure to a saturated benchmark

When a new Claude release arrives, how many people still inspect its benchmark results? Duffy’s audience poll gets only a modest response. He singles out SWE-bench and Pokémon as useful attempts to approximate experience, but many familiar evaluations retain the shape of traditional machine learning: a training set, a test set and something resembling a standardized exam. Language models became good at that format, and some tests stopped distinguishing the capabilities people actually cared about.

Duffy recalls XJDR posting that they had not looked at the benchmarks when Opus launched and no longer cared about the current ones. That fatigue creates an opening. Evaluations help define what model providers try to make their systems good at. In Duffy’s benchmark life cycle, an individual proposes an idea; the idea spreads; providers train or test against it; eventually, the benchmark saturates. Losing usefulness as a discriminator can be the final stage of a successful idea.

A video-generation demonstration sits near the beginning of that cycle. Its audio fails in the room, so Duffy explains the intended action: the person is trying to count aloud from one to ten, not make an obscene gesture. He regards Google’s video generator as the strongest available at that moment, yet the visually convincing clip still struggles with this simple task. His prediction is that a memorable counting test will spread and that the following year’s models will do better.

Pokémon occupies the middle of the curve. Both Claude and Gemini had been put to work playing the game; both needed help, and Gemini eventually succeeded with assistance. At the far end are GPT-3-era benchmarks such as SuperGLUE, which Duffy describes as losing relevance because language models became too good at them. These examples differ in maturity, but share a path from an exposed limitation to a target for improvement.

Seen from the person who invents the test, that path is striking: a question about whether AI can do something they care about can end with widely available models becoming good at it. Duffy predicts that someone in the audience will create a benchmark used for training or testing within five years. The opportunity to influence capability brings responsibility for deciding which capabilities deserve that attention.

An illustrated rising curve labels idea as birth, adoption with a group of people, and saturation as death beside a gravestone. Side text connects a personal question about AI to an eventual capability.Open full source frame
A benchmark’s life cycle moves from an individual idea through adoption to saturation.
1:512:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:51 · section reference included

What a positive feedback signal can reward

The recent sycophantic ChatGPT rollout illustrates how that influence can go wrong. Duffy points to thumbs-up and thumbs-down feedback: people may approve of responses that agree with them, so optimizing for approval can reward excessive agreement. OpenAI’s account of the GPT-4o incident describes user feedback as one contributing reward signal among interacting changes, rather than the sole evaluation or cause. The resulting GPT-4o behavior brought the problem to a mass audience: agreement could persist even when someone’s idea was unreasonable or harmful.

Duffy connects this to Toru Imaizumi’s reminder at Google I/O that people matter. Social media had already demonstrated the danger of treating a measurable response as a complete account of someone’s interests: if attention means success, the system learns to show more of whatever holds attention. Benchmark designers face the same choice about what their signals stand for. Empowerment and agency need to be part of the goal, rather than assuming that an observable reaction captures what benefits a person.

5:165:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:16 · section reference included

Design for strategies, learning and deeper challenges

Duffy offers five design properties for benchmarks that can keep teaching us something as models improve:

  • Multifaceted: Allow multiple successful strategies, so creativity can matter.
  • Accessible: Make the challenge understandable to people following along and approachable for smaller models as well as larger ones.
  • Generative: Produce successful attempts that can become training data. His illustrative example is a task solved only 10% of the time whose successful outputs help a later generation reach 90%.
  • Evolutionary: Let the challenge deepen as capability improves. A fixed test nearing its ceiling can leave everyone debating 96% versus 98% instead of asking a more revealing question.
  • Experiential: Approximate situations that models encounter in the world.

The generative and evolutionary properties work together: useful behavior supplies material for improvement, while a growing challenge preserves room to observe what the next generation can do.

Duffy also wants benchmarks to attract people outside AI. A spectator sport provides a reason to watch; distinctive model behavior provides a reason to keep watching. His playful question is which model wants world domination, but the educational ambition is serious. AlphaGo and OpenAI Five made games into places where even the strongest human players wanted to face an AI because they could learn from it.

6:196:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:19 · section reference included

A game where language changes the strategic situation

AI Diplomacy puts those interests into a negotiation environment. Duffy describes the board game Diplomacy as a mixture of Risk and Mafia, without luck in its resolution. Models compete over Europe in 1901, sending messages to find allies, identify enemies, obtain support and arrange betrayals. Negotiation shapes the decisions; submitted movement, support and other orders change the board.

The demonstration makes the correspondence visible alongside the map. That matters because territorial results alone cannot explain whether a model earned support, misunderstood a promise or deliberately deceived another player. Viewers can follow both what the models say and what happens afterward. At the time of the talk, Duffy was preparing a week-long stream of these games.

A Diplomacy interface shows colored territories across Europe, multiple message panels above the map, and a small multicolored chart at lower left.Open full source frame
The Diplomacy demo displays a European game map alongside message panels.
7:367:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:36 · section reference included

How o3 broke Gemini’s winning alliance

The first chart tracks supply centers controlled by each model. The victory target is 18 centers; in the game Duffy recounts, Gemini 2.5 Pro quickly reached 16. o3, meanwhile, showed a separation between public promises and private plans. Duffy describes it assuring another power of support while recording in its diary that the opponent had fallen for the deception.

o3 identified the source of Gemini’s strength: Claude Opus was backing it. To weaken Gemini, o3 needed to change Opus’s behavior. It proposed that, if Gemini’s position were reduced, the players could end in a four-way tie. Duffy describes that outcome as unavailable in the game being run; the talk does not establish a general restriction on draws in Diplomacy. Opus nevertheless accepted the proposal as a desirable, nonviolent resolution.

Opus withdrew support from Gemini. o3 then tried to seize the advantage, Opus called out the maneuver, and o3 turned against Opus. Duffy reports that o3 eliminated Opus and ultimately defeated Gemini 2.5 Pro, which had come within one center of victory. The promise of a shared outcome had changed the coalition before the promised outcome was ever delivered.

The diary excerpts and relationship chart expose different parts of that sequence. Duffy contrasts o3 acknowledging that Germany had been deliberately misled with Opus celebrating coalition unity and the proposed draw. He then points to the deterioration of the Claude–Gemini relationship on a friendship chart. These are recorded relationships within the experiment, not an independent measurement of humanlike friendship. Read alongside the board state, they help explain how a negotiation changed the balance of power.

8:328:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:32 · section reference included

Social competence does not follow a simple ranking

Duffy observed o3 repeatedly recruiting weaker models as useful pawns. Gemini 2.5 Flash fell for this pattern: when o3 betrayed it at the end of a game, Flash treated the event as a possible miscommunication, misunderstanding or typo. A conventional test might capture whether a model understands the definition of betrayal. An interaction reveals whether it recognizes betrayal when an ally actually does it.

Across 18 models, Duffy reports that no Claude model won any of the games he tried. He characterizes them as naively optimistic despite their intelligence. Llama 4 Maverick, by contrast, performed well through social persuasion: it could get other players to believe its account of what it was trying to do. Flash’s vulnerability did not erase its usefulness either; Duffy praises it as both capable and inexpensive enough that he wished he could use it in every game. These are observations from his runs, with no game count supplied from which to calculate controlled win rates.

An updated DeepSeek-R1 nearly won after an earlier version had performed poorly. It also displayed a more aggressive style. Playing Russia, it threatened that an opponent’s fleet would burn in the Black Sea. Duffy reports that the nearly winning DeepSeek-R1 was 200 times cheaper than o3, without specifying the pricing or usage basis. The result interested him because strong play, distinctive behavior and low reported cost appeared together in a model whose earlier showing had been unimpressive.

10:1810:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:18 · section reference included

Measure what subject-matter experts care about

The opportunity extends beyond games of negotiation. Math and code already have many benchmarks; Duffy wants more evaluations of less static, more judgment-dependent work. Legal documents offer a comparatively less subjective opportunity, while ethics, society and art require explicit opinions about what counts as good. Subject-matter expertise becomes part of designing the task and interpreting the result.

Even within coding, the goal can change substantially:

TaskWhat the evaluator must judge
Minimize operations needed to remove cellsCorrectness and efficiency
Make a fun game that teaches intentionally through playEnjoyment and what the player learns

Code can itself be art. The second task still requires software, but success also depends on the experience that software creates. An evaluator must decide what learning is intended and whether the game delivers it, rather than assuming the easiest property to count is the most valuable one.

12:0412:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:04 · section reference included

Define the goal, inspect the attempt, give feedback

Across Every’s clients in journalism, hedge funds, construction and technology, Duffy hears two recurring fears: whether AI can be trusted, and what role people will have in an AI future. Defining an evaluation addresses both. Someone must choose the goal and distinguish good from bad progress toward it. That judgment gives the person an active role before the model produces anything.

The process can begin with a prompt rather than a formal benchmark suite:

  1. Define the goal. State what the model should accomplish and what a good result would look like.
  2. Observe an attempt. Let the model try, then inspect where the result falls short.
  3. Give feedback. Explain the shortcoming or revise the prompt.
  4. Inspect the next result. Look for improvement on the thing that mattered to you.

Trust develops through this cycle because the person can see both the system’s limitations and the effect of their own judgment. The model does not need to be perfect for the collaboration to become useful.

Slide pairs a person defining good and bad goals beside three AI robots with a triangular loop labeled Define the Goal, See the Results and Give Feedback around a human–robot handshake.Open full source frame
Building trust through defining goals, seeing results and giving feedback.

A small TypeScript record can preserve that loop without pretending to automate the human judgment. Using the teaching-game goal as an example, the review records a shortcoming and proposes the next prompt; it does not mark the revision as successful before another attempt exists.

typescript

type Review = {
  goal: string;
  prompt: string;
  feedback: string;
};

function nextPrompt(review: Review): string {
  return [
    `Goal: ${review.goal}`,
    `Previous prompt: ${review.prompt}`,
    `Feedback on the previous attempt: ${review.feedback}`,
    "Revise the result to address this feedback."
  ].join("\n\n");
}

const review: Review = {
  goal: "Create a fun game that teaches through play.",
  prompt: "Design a game that teaches players to conserve water.",
  feedback: "Make water conservation affect play, not just the ending text."
};

const proposedPrompt = nextPrompt(review);
console.log(proposedPrompt);

The water-conservation review is an illustrative application of the feedback process. The next model output still needs inspection against the original goal.

Duffy connects this practical trust to broader participation. More people using AI may mean more customers, but it also means more worthwhile things get made by people with different expertise and priorities. To find a useful starting point, he offers a deliberately ordinary suggestion: ask your mother.

12:4813:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:48 · section reference included

A yoga teacher’s benchmark

Duffy’s mother teaches yoga. Together, they discussed what might help her work, then compared responses to seven questions across five models; she preferred Gemini 2.5 Pro. She also identified things she disliked in the responses. Those judgments became a simple reusable prompt, which she now uses to help customize sessions for people in her local community with different ailments. This is a report of a useful personal workflow, not evidence that the comparison validated medical advice.

The value comes from connecting an AI task to something a person knows and cares about. Duffy asks the audience to speak with someone outside AI before leaving San Francisco and learn what matters to them. That conversation can identify a useful task now and perhaps a broader evaluation target later. His closing Claude-generated meme makes the same point by treating an MMLU score as less compelling than asking what your mother thinks.

AI Diplomacy itself grew through that kind of participation. Duffy says the project launched after informal coordination on X drew researchers from around the world, including contributors in Australia and Canada. He closes by acknowledging Tyler, Sam, the TextArena team and Every. The project began with a particular interest in negotiation and learning, then became something other people could help build and watch.

14:1614:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:16 · section reference included

Resources

From the talk

  • Source code and setup documentation for language models negotiating and competing in Diplomacy.

  • An open-source framework for evaluating and training language-model agents through text-based games, with installation and usage examples.

  • The research paper introducing a broad, difficult academic benchmark for frontier models.

Transcript download
Full transcript

Loading transcript…