AI Engineer Summit 2025

How we scaled 500m AI agents in production with 2 engineers

Mustafa Ali· Senior Software Engineer, Method FinancialKyle Corbitt· Founder & CEO, OpenPipe18:44

Loading slides…

Read the talk

How we scaled 500m AI agents in production with 2 engineers

Method’s missing financial fields led from manual bank calls to GPT-4, then to a smaller model trained on production tasks to meet quality, latency, and cost requirements.

From a talk by Mustafa Ali and Kyle Corbitt

Before you start: Familiarity with LLM API calls, prompts, and the basic purpose of fine-tuning will help you follow the model-selection decisions.

Getting financial data that has no central API

How do you deliver financial information when the data your customers need is still difficult to access? Method aggregates liability data from hundreds of sources, including credit bureaus, Visa and Mastercard, financial institutions, and third parties. Its fintech, bank, and lender customers use that information for refinancing, loan consolidation, liability payments, and personal finance management. Kyle Corbitt introduces the case as Method scaling to more than 500 million agents, without defining the counting unit or time window. His company, OpenPipe, helps build, train, and deploy open-source models, using production signals to improve them over time.

The immediate problem was specific: customers already received balances and payment information, but also wanted an auto loan’s payoff amount or a mortgage’s escrow balance. Mustafa Ali and the Method team asked their data partners how to obtain those fields. They found no accessible central API. Direct bank integrations were an option, but Method expected them to take at least a couple of years. An early-stage company needed a route it could put into production much sooner.

Slide titled “Opaque Financial Data Access” mentions payoff amounts and escrow balances, lists missing central access and lengthy bank integrations, and asks “How do we go to market fast?”Open full source frame
Opaque financial data access: no central API, and bank integrations can take years.

Customers were already solving the problem through people. Offshore contractors called banks on behalf of the company and end consumer, authenticated, gathered the information, and sent it back for checking. The checked results then entered financial platforms, where they could support underwriting or appear directly to users.

That workflow scaled with headcount: one person could handle only one task at a time. Its synchronous nature made it slow, and human errors created another staffing requirement for checking results. The consequential failure was inaccurate financial information reaching a customer. Yet the process had recognizable API components—request, authentication, response, and validation. The missing interface was a way to turn unstructured information into reliable financial fields.

0:220:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:22 · section reference included

GPT-4 makes extraction work—and exposes the next bottleneck

The launch of GPT-4 gave Method a promising tool for parsing unstructured data. Its strength at tasks such as summarization and classification suggested that the information-gathering workflow could be automated. Method built an agentic workflow around it, then expanded the extraction tasks performed within a single API call to get more value from each expensive request. Controlled production tests worked well.

As traffic increased, the economics became harder to ignore. Mustafa reports a $70,000 bill for Method’s first month of GPT-4 production usage. Leadership was unhappy with the expense, but the value was substantial enough to keep the system running for at least a couple more months while the team searched for a cost solution.

Prompting also became a maintenance problem. A generally capable model still needed detailed financial instructions and examples. Prompts grew long and convoluted; fixing one scenario could break another, then fixing that regression could undo the first improvement. Method had no prompt versioning to help manage the changes.

The automated system began to reproduce the scaling problems of the manual one. Method attributed limited caching opportunities to variable responses and frequent prompt changes. That should be distinguished from provider prompt caching, which reuses shared input prefixes: variable outputs alone do not prevent prefix reuse, and the talk does not identify Method’s caching implementation. Slow baseline responses constrained concurrent operation, while hallucinations introduced errors that were difficult to catch. The team retained the workflow for the specific use cases where it worked well.

4:294:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:29 · section reference included

Measure the task before choosing the model

The problem had shifted from making extraction work to making the workflow reliable at volume. Method’s planning requirements were at least 16 million requests per day, at least 100,000 concurrent load, and latency below 200 milliseconds. These were targets for the proposed system, not measurements of simultaneously achieved performance. Buying GPUs and hosting a model were possibilities under consideration. OpenPipe joined the effort about a year before the talk, framing the decision around quality, latency, and cost.

The first step was to measure errors. Corbitt reports approximately 11% error for GPT-4o and 4% for o3-mini on Method’s tasks. Method could establish reference answers by having a human complete the information-gathering flow and identify the correct financial values. Comparing the agent’s final outputs with those values gave the team an outcome-based evaluation, rather than an assessment of whether an intermediate response sounded convincing. Zero error was not considered a feasible expectation.

Corbitt reports response latency of roughly one second for GPT-4o and five seconds for o3-mini on Method’s task. Reasoning duration depends on the work being done, so the useful measurement is one taken across the task diversity and concurrency expected in production. A fast response to a single convenient prompt does not establish the latency of the deployed workflow.

Cost introduced another surprise: although o3-mini had lower per-token pricing, Corbitt says it was slightly more expensive for Method because it generated more reasoning tokens and longer outputs. The comparison therefore looked like this:

DimensionGPT-4oo3-mini
Extraction qualityMore errorsFewer errors
Response latencyFasterSlower
Cost for Method’s taskLowerSlightly higher

Per-task cost matters more than the price of an individual token. A model can charge less per token and still cost more to complete the same job.

Three bar charts compare o3-mini and GPT-4o. o3-mini has lower error rate but higher average latency and cost per 1K calls; all charts indicate lower is better.Open full source frame
The Inference Trifecta compares error rate, average latency, and cost per 1K calls.

Once a proof of concept works, Corbitt recommends keeping simple Python scripts for evaluating quality, latency, and cost as new models arrive. For example, a summary over recorded runs can keep those dimensions separate. Here, each row represents a completed extraction with a human-checked expected result, measured latency, and recorded cost:

python

from statistics import mean


def summarize(runs):
    if not runs:
        raise ValueError("At least one completed run is required")

    errors = sum(
        run["actual"] != run["expected"]
        for run in runs
    )
    return {
        "runs": len(runs),
        "error_rate": errors / len(runs),
        "mean_latency_ms": mean(
            run["latency_ms"] for run in runs
        ),
        "cost_per_1000_calls_usd": 1000 * mean(
            run["cost_usd"] for run in runs
        ),
    }

This example uses exact equality for the final extracted result. The application must define what constitutes a correct result and capture representative runs; the summary function only aggregates those observations.

7:497:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:49 · section reference included

Set thresholds for the whole application

With the baseline measured, the next question was what performance the application actually required. Method already applied downstream checks to extracted numbers: were they plausible, and did they resemble values seen before? Corbitt reports that around 9% model error met Method’s needs when combined with those downstream checks. That is a model-stage tolerance, not a claim that delivering incorrect financial data at that rate was acceptable. The checks themselves were fallible, so increasing the incoming error rate would allow more mistakes through.

Latency requirements came from the interaction. Method’s agent had to respond quickly enough to move through a real-time information-gathering flow, creating a hard cutoff. Corbitt contrasts that with two other workloads:

  • Background batches: Some customers can wait days for a result.
  • Human-facing voice: Some customers cannot tolerate response latency above 500 milliseconds.

High request volume made cost especially consequential for Method. Quality, timing, and price therefore needed application-specific thresholds, rather than a single model ranking detached from how the result would be used.

Neither candidate met all three requirements. Corbitt identifies GPT-4o’s error rate and cost as problems; o3-mini’s cost and especially its latency also prevented deployment under the desired conditions. That was the point at which a custom model became worth considering. Fine-tuning requires more time and engineering investment than prompting, so benchmark prompted models first, then tune when the measured gap justifies the work.

11:2211:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:22 · section reference included

Train a smaller model on production inputs

The tuned model improved on GPT-4o’s error rate and cleared Method’s quality threshold. Producing the training data did not require manually labeling every example. A stronger reasoning model could generate answers for inputs already encountered in production, reducing the manual work that had previously made this approach harder. The procedure was straightforward:

  1. Collect inputs from the production workload.
  2. Generate outputs for those inputs with o3-mini.
  3. Train a smaller model on the resulting input-output pairs.

This moves the more expensive model’s work into training-data generation, allowing the smaller model to learn the specific task.

The deployed model was Llama 3.1 8B, with eight billion parameters; the talk does not specify its starting checkpoint. Its quality did not quite match the o3-mini teacher, but it came close enough to outperform GPT-4o on this workload and satisfy Method’s needs. Corbitt says models of that size or smaller meet the quality requirements of most OpenPipe customers, while emphasizing that each application still needs its own benchmark.

The smaller model also made low-latency serving easier. Corbitt attributes that to fewer calculations, including fewer sequential calculations through model layers. A further option is to deploy the model alongside the application that calls it, removing the remote network round trip. Method did not need that co-location step.

The same reduction in model size lowered serving cost. Corbitt reports that the tuned model comfortably beat Method’s cost requirement, easing the unit-economics concern created by the larger models. Fine-tuning still added engineering work, but it changed the available trade-off: the team could obtain the required task quality from a model that was cheaper and easier to serve quickly.

14:1314:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:13 · section reference included

Why two engineers could do it

Mustafa explains the title’s two-engineer claim through the scope of the work. The team identified a specific use case and fine-tuned an inexpensive model that delivered the speed it needed. Existing GPT production data supplied the training material, so there was no separate hunt for a dataset. The earlier production system had created both customer value and the examples needed to improve its economics.

The team did not need to buy its own GPUs. It did need openness and patience from engineering and leadership while the agent’s responses improved enough for production. Deploying an agent involves iteration on behavior, not just shipping a feature and assuming its responses will already meet the application’s needs. That organizational willingness to keep refining the system was part of making the technical approach work.

Slide titled “Lessons Learned at Method” lists “The cheapest model just works,” “Don’t buy your own GPUs,” and the need for openness and patience when productionizing AI agents.Open full source frame
Lessons learned at Method: inexpensive models, no owned GPUs, and patience in productionizing agents.

Mustafa closes with a career joke that fits the work the team had just described: “If you're in SWE, pivot to AIE.”

17:1017:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:10 · section reference included

Resources

From the talk

  • Code and setup instructions for request capture, dataset management, fine-tuning and evaluation. The README currently notes paused development of the open-source version.

  • Explains input-prefix reuse, original model eligibility and cache-usage reporting at the October 2024 launch.

Updates since the talk

Transcript download
Full transcript

Loading transcript…