AI Engineer World's Fair 2025

Analyzing 10,000 Sales Calls with AI in 2 Weeks

Charlie Guo· AI engineer, Pulley9:49

Loading slides…

Read the talk

Analyzing 10,000 Sales Calls with AI in 2 Weeks

Charlie Guo’s customer research project at Pulley shows how model selection, traceable extraction, prompt caching, and a reusable interface turn a transcript archive into a working research tool.

From a talk by Charlie Guo

The two-week analysis challenge

How many sales calls can you listen to and take notes on in a day? At 30 minutes per call, an eight-hour day without breaks fits 16 calls. Spend every waking hour listening, with only eight hours reserved for sleep, and that becomes 32 calls a day, or 224 per seven-day week.

Charlie Guo faced a different scale: Pulley’s CEO wanted 10,000 sales calls analyzed in two weeks to refine the company’s ideal customer profile, or ICP. Guo describes a project that a single AI engineer could complete in a fortnight—work that previously would have required a dedicated team over several weeks. The challenge was to turn conversations into trustworthy customer research quickly enough to inform a business decision.

Slide reading “10,000 Calls” and “2 Weeks,” with the presenter inset on the right.Open full source frame
10,000 calls, two weeks.
0:040:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:04 · section reference included

From a broad ICP to specific customer evidence

Pulley’s existing ICP was venture-backed startups. That was a useful guide for product development, but too broad for marketing. A persona such as CTO of an early-stage, venture-backed crypto startup gives a marketer more to work with than founder. Understanding that specificity requires listening to customers; Pulley already had thousands of hours of sales representatives doing exactly that.

A manual research process would repeat the same sequence for each call:

  1. Download the transcript and read the conversation.
  2. Decide whether the conversation matches the target persona.
  3. Scan hundreds or thousands of lines for useful insights.
  4. Compile notes and reports while retaining citations for later verification.

At 16 calls per day, 10,000 calls require 625 eight-hour workdays—nearly two years only if someone works every day. Even finishing the reading would leave the synthesis problem: remembering enough of an entire library to write one book report.

The familiar approaches each sacrificed something essential:

ApproachStrengthLimitation
Manual analysisHigh-quality interpretationCannot scale to this workload
Keyword analysisFast and cheapMisses context and nuance

Large language models offered a way to recognize patterns in unstructured conversations without reducing every question to a keyword match. But producing useful research still required solving several connected problems: choosing a model, controlling unsupported claims, preserving evidence, and paying for the resulting workload.

1:011:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:01 · section reference included

Choose the model against consequential errors

At the time of the project, GPT-4o and Claude 3.5 Sonnet were the most capable models available to Guo’s team, but also the slowest and most expensive. Smaller, cheaper models were attractive until the team examined their classifications.

Two failures exposed the difference between recognizing words and establishing facts:

  • Company classification: A sales representative mentioned the platform’s blockchain features, and a smaller model classified the prospect as a crypto company. The feature belonged to the sales pitch; it did not establish the customer’s business.
  • Person classification: A model labeled a prospect as a founder despite no supporting evidence in the transcript.

These false positives threatened the purpose of the project. A polished report built from incorrect customer identities would produce a bad analysis. The team chose Claude 3.5 Sonnet because its hallucination rate was acceptable for their needs, although Guo gives no numerical error rate. The cost of an unsupported classification mattered more than the price of an individual request.

3:123:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:12 · section reference included

Make extracted findings traceable

Model selection was only the first layer of reliability. Guo’s team started with raw transcripts, enriched them through retrieval-augmented generation using third-party and internal sources, and used prompting techniques, including chain-of-thought prompting, to improve the results. The model therefore had additional context to help interpret the conversation rather than relying on the transcript alone.

The next layer was structured JSON output, used where possible to generate citations. A useful output contract keeps a finding and its evidence together. For example, the earlier blockchain failure can be represented with an explicitly unknown customer classification rather than an unsupported positive label:

json

{
  "transcript_id": "call-001",
  "company_sector": null,
  "evidence": [
    {
      "line_id": "line-042",
      "speaker_role": "sales_rep",
      "text": "Our platform includes blockchain features."
    }
  ],
  "finding": "The sales representative describes a platform feature; the prospect's sector is not established."
}

This illustrative record makes the evidence boundary concrete: the quoted line supports a statement about the sales pitch, while company_sector remains unknown. The project used application-generated citations from JSON; it should not be read as a demonstration of Anthropic’s later native citations API, whose launch is recorded in the Claude Platform release notes.

The resulting system extracted company details and customer insights with a verifiable trail back to the original transcripts. That trail made it possible to inspect the basis of a finding instead of accepting the model’s answer on authority. But richer analysis and the effort to keep errors low increased the cost substantially.

4:134:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:13 · section reference included

Reduce repeated input work and split outputs

The workflow often reached the 4,000-token output limit Guo encountered with Claude 3.5 Sonnet. Completing one transcript’s analysis could therefore require several requests. Two features that were experimental at the time addressed different sources of repeated work.

Prompt caching addressed repeated input. The team reused the same transcript to extract metadata and then insights, so caching let later requests reuse that transcript content. Guo cites reductions of up to 90% in cost and up to 85% in latency for prompt caching. Those figures match the ceilings advertised in Anthropic’s prompt-caching announcement, rather than a supplied measurement of this project’s overall performance. Cache writes carry a premium, and the announcement’s latency example measures time to first token; the benefit depends on reusing the long prompt.

Extended outputs addressed incomplete responses. An experimental feature flag doubled the available output capacity, allowing complete summaries in a single pass instead of splitting the analysis across multiple turns. This expanded the amount the model could generate, not its input context window. Historically, Anthropic introduced the 8,192-token output beta in July 2024 and removed the beta-header requirement in August; the flag described here belongs to that earlier API period.

“Cutting Costs” slide lists prompt caching with 90% cost reduction and 85% latency reduction, plus longer outputs up to 8K tokens (2x default) and single-pass analysis.Open full source frame
Prompt caching and longer outputs support cheaper, single-pass analysis.

Guo reports that the optimizations reduced the analysis cost from $5,000 to $500 and delivered results in days instead of weeks. That is his project-level account, without a billing or runtime breakdown. The engineering mechanisms are distinct: caching reduces repeated processing of the transcript, while longer outputs reduce the need for additional requests to finish a summary.

5:185:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:18 · section reference included

The research infrastructure reaches other teams

The larger surprise was how widely the work became useful. The initial customer was the executive team, which wanted insights that Guo says would normally require tens, if not hundreds, of hours to produce. Once the underlying transcript workflow existed, other teams could use it for their own work.

  • Marketing: Retrieved customer quotes for branding and positioning exercises.
  • Sales: Automated transcript downloads using the system. Guo reports that automating transcript downloads saved the sales team dozens of hours every week.

The value extended beyond the generated research: even the surrounding data-access machinery removed recurring work.

Slide titled “Cross-Organizational Impact” lists Leadership, Marketing, and Sales beside the presenter.Open full source frame
Cross-organizational impact spans leadership, marketing, and sales.

Teams also began asking questions they previously would not have considered because manual analysis was too daunting. Making the corpus accessible changed which research questions were practical to pursue, turning an accumulation of unstructured conversations into a shared source of evidence.

6:226:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:22 · section reference included

Build the system around the model

Model capability mattered: in Guo’s experiments, Claude 3.5 Sonnet and GPT-4o handled tasks the other tested models could not. But the final choice combined accuracy with prompt caching. Model fit depended on both the errors the project could tolerate and the economics of repeatedly analyzing long transcripts.

The surrounding software mattered just as much. Structured JSON, good database schemas, and the overall architecture produced significant gains. AI engineering means building an effective system around the model: its outputs need to fit the application’s data structures and workflows, and its evidence needs to remain available to the people using the results. Guo’s lesson is to integrate the model thoughtfully into the system rather than add it as an afterthought.

7:187:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:18 · section reference included

Make customer research a repeatable operation

The project did not end with a report or white paper. The team built an interface around the analysis with search, filters, and exports. Those features let colleagues find relevant material and carry it into their own work, making a simple, flexible tool useful across the company.

The purpose was to augment human analysis by removing the bottlenecks that prevented it. Once downloading, reading, and organizing the corpus no longer consumed the entire research budget, people could investigate questions that had previously been out of reach. Faster processing enabled a broader range of work.

That leaves a practical question for any organization: what customer data is already available but rarely examined? Sales calls, support tickets, product reviews, user feedback, and social media interactions all contain potential evidence. Guo’s closing challenge is to use the available tools and techniques to make those neglected sources accessible—and begin turning the accumulated data into something people can use.

8:198:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:19 · section reference included

Resources

  • Charlie Guo's earlier account of the sales-call analysis project, including model selection, transcript caching, citations and the interface built for colleagues.

  • Anthropic explains prompt reuse, cache pricing and example cost and time-to-first-token improvements for long inputs.

  • Dated API history covering Claude 3.5 Sonnet's longer outputs, prompt caching and subsequent citation support.

Transcript download
Full transcript

Loading transcript…