AI Engineer World's Fair 2024
How Zapier Builds AI Products and Features With the Help of Braintrust
Loading slides…
Read the talk
How Zapier Builds AI Products and Features With the Help of Braintrust
Zapier’s path from prompt-generated workflows to an interactive Copilot shows how product criteria, regression tests and tracing make model changes measurable.
From a talk by Ankur Goyal and Olmo Maldonado
What makes a generated workflow correct?
How do you know whether an AI-generated workflow will actually work? That question brings product requirements and evaluation tooling together. Ankur Goyal describes building evaluation tools at Impira and later while leading the AI team at Figma, before founding Braintrust. Zapier became its first users, and the collaboration helped shape the platform.
At the time of the talk, Olmo Maldonado reports that Zapier connected over 7,000 apps and ran over 10 million tasks per day. That task volume describes Zapier overall, not AI-only usage. Its low-code and no-code interface makes integrations accessible, but the workflows still need to execute reliably.
Two Zapier AI products take on different parts of that work: AI Zap Builder generates workflows from prompts, while Zapier Copilot supports an interactive approach inside the editor. Maldonado also introduces Central, the historical bot framework for connecting bots to users’ apps, though the detailed examples focus on Builder and Copilot.
Braintrust supplies three connected capabilities: evaluations to measure behavior, observability to inspect what happened, and prompt development tools to change it. Goyal emphasizes that evaluation and observability form a continuum: measuring an output and understanding the execution that produced it belong in the same development loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a prompt-to-Zap demo into product criteria
Zapier’s process starts with getting a prototype to users early, learning from mistakes and making adjustments through evaluations. AI Zap Builder makes the initial interaction look straightforward: enter a prompt, let it propose a Zap, then click Try It to create the workflow. It can also handle field mapping. The difficult question comes after generation: how well does that output deliver the automation the customer wanted?
The long tail of integrations makes a handful of successful examples insufficient. Zapier involved product managers in defining the evaluation requirements, treating correctness as both a product and an engineering problem. Its P0 priorities made the intended behavior concrete:
- Triggers and actions: Choose the expected operations and put steps in the correct positions.
- Priority integrations: Support the top 25 apps correctly.
- Internal workflow features: Include Zapier’s Paths and Filters in the evaluation suite.
Product priorities determine what the suite must measure. The objective is not merely to produce a plausible-looking workflow; it is to cover the behaviors the product promises.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Seed, run and grade the suite
Zapier built an in-house framework with Braintrust around a repeatable sequence:
- Seed coverage with synthetic data from corporate accounts.
- Load the examples from datasets hosted in Braintrust.
- Execute the suite in CI and through manual runs. A custom runner exercises the AI providers in what Maldonado describes as a load-test-like process.
- Grade and report the outputs using both logic-based and LLM-based custom graders, then act on the failures.
The graders connect execution back to the product criteria. They provide multiple views of whether the generated workflow does what the team intended, rather than reducing every requirement to a single undifferentiated judgment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make regressions visible and actionable
Frequent runs shorten the interval between introducing a regression and discovering it. In the AI Zap Builder project shown, developers can inspect individual inputs and outputs, compare a run with previous experiments, and see which grader scores improved or declined. The comparison view makes the product’s most important criteria visible so the team can decide where to intervene.
Zapier expanded from seven unit tests run manually by developers to over 800 tests run with merge requests and continuously, with alerts on regressions. Maldonado reports nearly 300% improvement in accuracy after introducing this process. He explicitly cautions that this does not mean 100% accuracy; the talk supplies neither the baseline nor the calculation needed to translate the improvement into an absolute accuracy figure. The remaining failures still require work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the move from generation to conversation
Positive customer feedback did not remove the limits of a single-shot interface. The next iteration, Zapier Copilot, lets the user work progressively through chat inside the editor. The demonstration starts with the same prompt used for Builder, then goes further: Copilot tests steps and configures fields as the workflow develops. Maldonado says the displayed demo is not sped up.
That richer interaction introduces a new debugging problem. With an agent framework calling multiple tools, the team could no longer easily identify the critical path or determine which part needed improvement. A final output alone does not explain where time was spent or where a decision went wrong.
Braintrust’s tracing breaks the request into smaller operations. Developers can inspect a chat completion’s inputs, outputs, token usage and response time, then use that detail to investigate Copilot’s behavior. The expanded trace shown exposes an individual tool call inside the larger execution. Maldonado also describes using a playground to iterate on inspected behavior, although the playground itself is not demonstrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare models inside the application
Copilot initially used GPT-3.5 Turbo to get to market quickly. As the team tested alternatives, it manually assembled a comparison view showing model performance across different tools, including how that performance changed when the tools changed. The team selected GPT-4 Turbo for its message router, accepting slower responses in exchange for greater confidence in routing accuracy.
The next attempt to improve speed introduced GPT-4o—and the evaluation suite detected regressions. Recovering the earlier behavior required prompt changes and changes to parameters in the OpenAI calls. This was a migration problem in Zapier’s particular application: a model replacement changed how the existing instructions and tool setup behaved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow falling scores into the prompt
Before the GPT-4o switch, most evaluation scores were at least 80%; Maldonado reports that the affected scores fell below 80% afterward. The team wanted the new model’s speed and cost benefits without accepting that loss in quality. The investigation shown was the work of his colleague Maggie Cody.
The inspected comparison contained 22 regressions associated with unwanted answers or apparent failures to follow system instructions. Maldonado describes the behavior as the model seeming to forget the system prompt. That is a description of the observed responses, not evidence that the instructions were absent from the request. The team’s diagnosis was that its prompts had become too tailored to GPT-3.5 Turbo.
The response was to undo some of that accumulated prompt engineering. Rather than adding more elaborate demands, the team relaxed the instructions and reran evaluations; Maldonado reports recovering the previous score levels. The displayed prompt diff makes the change tangible: longer instructions are removed and shorter instructions are added. A prompt tuned around one model’s behavior is part of what must be reevaluated when the model changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover quality, then adopt the faster model
Rapid trial and error also led to a change in tool selection. Maldonado describes moving away from deprecated function-calling usage to tool_choice. The precise distinction in the OpenAI Chat Completions reference is that functions is replaced by tools, while function_call is replaced by tool_choice; the talk compresses those separate parameter changes. Its demonstrated setting is tool_choice: "auto", with "required" still a planned experiment.
The settings have different behavioral contracts:
| Setting | Model behavior | Status in this account |
|---|---|---|
auto | May answer directly or call tools | Used in the revised setup |
required | Must make one or more tool calls | Planned experiment |
That distinction matters when evaluating unwanted direct answers. Requiring a tool call changes the allowed behavior, but it was not yet a demonstrated recovery mechanism here. After the prompt and API changes, the team compared experiments against earlier runs and saw most scores improve.
Zapier subsequently adopted GPT-4o while continuing to iterate. Maldonado reports that streaming Copilot latency fell from around 14 seconds before adoption to three seconds afterward. The talk does not specify whether that measures first output or a complete response. The result closes the development loop: evaluations exposed the quality regression, individual failures guided revisions, and comparisons gave the team a basis for adopting the faster model without simply accepting the initial score drop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore Zapier's current AI automation products and connected-app capabilities.
The original announcement of Zapier's bot workspace, including connected apps, live data, and automated behaviors.
Updates since the talk
- Introducing Zapier AgentsArticle
The announcement explaining Central's transition to Zapier Agents.
Current guides to tracing, evaluations, datasets, prompt development, and playgrounds.
Request parameters for Chat Completions, including tool definitions, tool selection, and deprecated function-calling fields.
Full transcript
Loading transcript…