AI Engineer Code 2025

Your Support Team Should Ship Code

Lisa Orr· Group PM, Zapier16:06

Loading slides…

Read the talk

Your Support Team Should Ship Code

Zapier’s Scout turns fresh support issues into proposed code fixes by gathering context, coordinating tools, and keeping support and engineering in the review loop.

From a talk by Lisa Orr

Before you start: Familiarity with APIs, support tickets, and merge requests will help; no prior knowledge of Scout or MCP is required.

A canyon of changing APIs

After 18 days rafting more than 200 miles through the Grand Canyon without internet or cell service, Lisa Orr returned to work with a useful analogy. Wind, water, and time continually reshape the canyon. At Zapier, third-party API changes continually reshape the maintenance work required to keep integrations reliable. Orr calls this app erosion.

At the time of the talk, Orr reports that Zapier had more than 8,000 integrations built on third-party APIs. The company had been around for 14 years, and some integrations were that old. Changes and deprecations could undermine previously working behavior at any time; maintenance had no finish line.

Slide pairs a Grand Canyon rafting photo with App Erosion bullets: 14 years; API Changes, Deprecations; Creates reliability issues; Never stops.Open full source frame
Zapier’s app erosion: API changes and deprecations create ongoing reliability issues.

Imagine the integrations as layers in the canyon walls and support as the river running between them, encountering the places where things are breaking. Tickets were arriving faster than the organization could resolve them. The growing backlog meant unreliable integrations, poor customer experiences, and even churn. Zapier started two parallel experiments: let support fix bugs as well as triage them, and investigate whether AI could make those fixes faster.

0:380:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:38 · section reference included

Give support a bounded path to shipping fixes

The support experiment began two years before the talk. App erosion was a major source of bugs passed from support to engineering, so there was a clear maintenance need. There was also interest from support: many team members wanted engineering experience, and some were already helping maintain apps unofficially. The experiment made that work an authorized responsibility.

Zapier started with three boundaries:

  • Four target apps: concentrate the initial work on a small, defined set of integrations.
  • Engineering review: have engineering review every merge request submitted by support.
  • App fixes only: keep the work focused on repairing integrations.

Support could now propose and develop fixes within a defined scope, with engineering remaining part of the approval path.

Experiment 1 slide lists three guardrails beside a flowchart from issue detection through ticket generation and triage to submitting a fix for engineer review or assigning a team.Open full source frame
Support starts with four apps, engineering review of merge requests, and a focus on app fixes.
2:372:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:37 · section reference included

The expensive work before writing code

The second experiment became Scout, the code generation project Orr had been leading over the preceding couple of years. Discovery started with doing the work: Orr shipped app fixes, and the team shadowed engineers and support members as they repaired integrations. They mapped the phases, pain points, and time spent throughout the process.

A major cost appeared before anyone could confidently write a fix: gathering and understanding context. A person might need third-party API documentation, reports of an emerging bug elsewhere on the internet, an external solution someone had already found, internal information, and logs. Finding those sources was only part of the task. The person also had to understand how they fit together and what they implied about the failure.

Scout initially addressed individual pain points with APIs. Some required language models; others were ordinary retrieval tools.

ToolMechanism
DiagnosisGather and curate context, then use an LLM to formulate a diagnosis
Unit test generatorUse an LLM to generate tests
Test case finderUse a search query to retrieve relevant test cases

The distinction matters: useful assistance included both generating new material and finding existing evidence. There was no need to make every operation an LLM call.

The team built many APIs and exposed them through an internal web playground called Autocode. Engineers were asked to visit it, try the tools, and provide feedback. But the tools were not embedded in the app-fix process. People already disliked opening many pages to assemble context; Autocode added another window to visit. Engagement was limited.

The broad API effort also spread the Scout team thin. Meanwhile, Cursor launched and gained strong adoption inside Zapier, making some of Scout’s tools unnecessary. The first phase had produced useful capabilities, but neither their availability nor their number ensured that people would use them.

3:293:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:29 · section reference included

Bring Diagnosis to the ticket, then tools to the IDE

Diagnosis was the exception. Support valued having the context assembled and curated before starting a fix, enough to request a Zapier integration for the Autocode APIs. They used it in the existing Zap that created a Jira ticket from a support issue. The ticket now arrived with Diagnosis included. The useful output appeared where support already worked.

That experience made workflow placement central to the next phase. With Model Context Protocol (MCP), the team could expose Scout’s API tools inside Cursor. Engineers could call them while working in the IDE, and Orr reports that builders using Scout MCP spent less time leaving that window.

Embedding the tools exposed another problem: Diagnosis took a long time to run. Its context gathering and recommendation were valuable, but waiting synchronously while working on a ticket in an IDE was frustrating. The team also struggled to keep up with customization requests. When the public Zapier MCP product became available, some engineers used it to meet needs that Scout’s tools did not address, leaving certain Scout tools at a dead end.

Adoption remained scattered. Some engineers used no Scout tools; others used only a few from the suite. The team’s next hypothesis was that more value would come from tying the tools together. Making each tool available still left the builder responsible for choosing the tools and coordinating the work across stages.

6:426:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:42 · section reference included

From a tool collection to a proposed fix

Scout Agent put that orchestration under the team’s control. It would run Diagnosis against a ticket, then pass the resulting context to a code generation tool that produced a merge request. This connected the investigation directly to a proposed code change.

Zapier had several integration teams handling app fixes of different levels of complexity. For the agent’s first customer, the team chose support: it was already handling small, emergent bugs arriving fresh from the queue. The experiment in giving support permission to ship code and the experiment in AI-assisted maintenance now converged.

The resulting workflow has an explicit gate before code generation and human review afterward:

  1. Categorize the submitted issue. Establish what kind of problem support has encountered.
  2. Assess fixability. Not every support issue is something Scout can fix.
  3. Generate a merge request if judged fixable. Support first picks up the ticket with a proposed change already attached.
  4. Review and test against the customer’s need. Support checks whether the proposed solution actually addresses the problem.
  5. Request adjustments in GitLab when needed. Scout takes another pass using the feedback.
  6. Submit the merge request for engineering review. Support’s acceptance of the proposal does not remove the engineering review boundary.

The automated output is a proposed fix ready for scrutiny, not an automatically approved change.

9:069:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:06 · section reference included

Run the work through Zaps, Jira, and GitLab

Scout runs through multiple Zaps embedded in support’s existing automations. Diagnosis runs first and posts its results to Jira, including the issue category and whether the issue appears fixable. A fixable result then triggers a GitLab CI/CD pipeline with three phases: plan, execute, and validate.

Those phases can be expressed as a GitLab stage declaration:

yaml

stages:
  - plan
  - execute
  - validate

This captures the phase ordering Orr describes; the jobs that perform Scout’s work sit within those phases. The pipeline uses Scout MCP, bringing the APIs built a year earlier into a coordinated process. Orr also identifies a component as “Cursor SDK.” That historical name should not be equated with today’s @cursor/sdk: the public SDK beta was announced in April 2026, after this talk, and Scout’s exact interface is not identified here.

Once the merge request is ready, the workflow attaches it to Jira for support to pick up. Context collection and the first proposed code change have therefore happened before support begins reviewing the ticket’s solution.

How we run Scout slide lists posting diagnosis to Jira, generating an MR through GitLab CI/CD using Scout MCP and Cursor SDK, and attaching the MR to Jira, beside a branching workflow screenshot.Open full source frame
Scout’s Zap-triggered workflow connects Jira to a GitLab CI/CD pipeline with Plan, Execute, and Validate phases.

The latest addition at the time of the talk shortened the revision loop. If support wanted changes, they could chat with Scout Agent inside GitLab. That feedback launched another pipeline pass and produced a new merge request, avoiding the need to pull the code into an IDE, make local edits, and push it back just to request a tweak.

11:0511:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Evaluate the decisions separately from the code

The team asks three separate questions about Scout Agent: Was the categorization correct? Was the issue actually fixable? Was the code fix accurate? These distinguish the decisions that route work into the pipeline from the quality of the change it produces.

Orr reports 70–75% accuracy for the two existing evaluations, categorization and fixability. She does not give separate scores for those tasks, a sample size, or a code-fix accuracy result. As the team processes more tickets and receives feedback, those examples become test cases for improving Scout. The evaluation set grows out of the work the system is actually handling.

12:3912:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:39 · section reference included

More proposed fixes, less searching for work

Orr reports that Scout generates 40% of the support team’s app fixes. That share concerns support’s app fixes, rather than all support tickets or all engineering work.

She describes two throughput changes: some support members doubled from one to two tickets per week; in the following account, support progressed from one–two fixes per person per week to three–four with Scout. The relationship between those baselines is not explained, so they should remain separate observations rather than a single calculated productivity gain.

There was also a change in how people found work. Scout placed potentially fixable tickets directly into the triage flow, reducing the friction of searching the backlog for something to take on. The system was helping surface a candidate task as well as prepare a candidate solution.

Scout’s impact on app erosion slide quotes support saying that placing potentially fixable tickets in the triage flow takes away friction when finding work in the backlog.Open full source frame
Support feedback credits Scout with reducing friction in finding fixable backlog tickets.

Engineering benefited too. An engineering manager’s feedback was that the tool allowed the team to stay focused on more complex work. Support’s expanded role absorbed some of the smaller app fixes while preserving engineering involvement in review.

13:0613:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:06 · section reference included

Keep the fix close to the customer

Support brings more than additional capacity to code generation. It has three advantages that affect the quality of a fix:

  • Proximity to customer pain: support is close to the context needed to understand what is wrong and what a useful solution would do.
  • Fresh evidence: troubleshooting happens while the issue is active. Context is current and logs are still available; after months in an engineering backlog, some of that evidence may be gone.
  • Customer-specific validation: a behavior change can help some users without solving the problem for the customer who reported it. Support can judge the proposed change against that customer’s actual need.

These strengths explain why support is well placed both to initiate the work and to test the generated proposal.

The experiment also created a path into engineering. Some of the support team members who participated had become engineers by the time of the talk. Giving people a bounded opportunity to ship fixes changed not only who handled the maintenance queue, but also what work those people could go on to do.

14:1014:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:10 · section reference included

Resources

From the talk

Updates since the talk

  • The April 2026 release introduces programmatic Cursor agents through the @cursor/sdk TypeScript package.

Transcript download
Full transcript

Loading transcript…