AI Engineer World's Fair 2025
Analyzing 10,000 Sales Calls with AI in 2 Weeks
Loading slides…
Read the talk
Analyzing 10,000 Sales Calls with AI in 2 Weeks
Charlie Guo’s customer research project at Pulley shows how model selection, traceable extraction, prompt caching, and a reusable interface turn a transcript archive into a working research tool.
From a talk by Charlie Guo
The two-week analysis challenge
How many sales calls can you listen to and take notes on in a day? At 30 minutes per call, an eight-hour day without breaks fits 16 calls. Spend every waking hour listening, with only eight hours reserved for sleep, and that becomes 32 calls a day, or 224 per seven-day week.
Charlie Guo faced a different scale: Pulley’s CEO wanted 10,000 sales calls analyzed in two weeks to refine the company’s ideal customer profile, or ICP. Guo describes a project that a single AI engineer could complete in a fortnight—work that previously would have required a dedicated team over several weeks. The challenge was to turn conversations into trustworthy customer research quickly enough to inform a business decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a broad ICP to specific customer evidence
Pulley’s existing ICP was venture-backed startups. That was a useful guide for product development, but too broad for marketing. A persona such as CTO of an early-stage, venture-backed crypto startup gives a marketer more to work with than founder. Understanding that specificity requires listening to customers; Pulley already had thousands of hours of sales representatives doing exactly that.
A manual research process would repeat the same sequence for each call:
- Download the transcript and read the conversation.
- Decide whether the conversation matches the target persona.
- Scan hundreds or thousands of lines for useful insights.
- Compile notes and reports while retaining citations for later verification.
At 16 calls per day, 10,000 calls require 625 eight-hour workdays—nearly two years only if someone works every day. Even finishing the reading would leave the synthesis problem: remembering enough of an entire library to write one book report.
The familiar approaches each sacrificed something essential:
| Approach | Strength | Limitation |
|---|---|---|
| Manual analysis | High-quality interpretation | Cannot scale to this workload |
| Keyword analysis | Fast and cheap | Misses context and nuance |
Large language models offered a way to recognize patterns in unstructured conversations without reducing every question to a keyword match. But producing useful research still required solving several connected problems: choosing a model, controlling unsupported claims, preserving evidence, and paying for the resulting workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the model against consequential errors
At the time of the project, GPT-4o and Claude 3.5 Sonnet were the most capable models available to Guo’s team, but also the slowest and most expensive. Smaller, cheaper models were attractive until the team examined their classifications.
Two failures exposed the difference between recognizing words and establishing facts:
- Company classification: A sales representative mentioned the platform’s blockchain features, and a smaller model classified the prospect as a crypto company. The feature belonged to the sales pitch; it did not establish the customer’s business.
- Person classification: A model labeled a prospect as a founder despite no supporting evidence in the transcript.
These false positives threatened the purpose of the project. A polished report built from incorrect customer identities would produce a bad analysis. The team chose Claude 3.5 Sonnet because its hallucination rate was acceptable for their needs, although Guo gives no numerical error rate. The cost of an unsupported classification mattered more than the price of an individual request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make extracted findings traceable
Model selection was only the first layer of reliability. Guo’s team started with raw transcripts, enriched them through retrieval-augmented generation using third-party and internal sources, and used prompting techniques, including chain-of-thought prompting, to improve the results. The model therefore had additional context to help interpret the conversation rather than relying on the transcript alone.
The next layer was structured JSON output, used where possible to generate citations. A useful output contract keeps a finding and its evidence together. For example, the earlier blockchain failure can be represented with an explicitly unknown customer classification rather than an unsupported positive label:
json
{
"transcript_id": "call-001",
"company_sector": null,
"evidence": [
{
"line_id": "line-042",
"speaker_role": "sales_rep",
"text": "Our platform includes blockchain features."
}
],
"finding": "The sales representative describes a platform feature; the prospect's sector is not established."
}
This illustrative record makes the evidence boundary concrete: the quoted line supports a statement about the sales pitch, while company_sector remains unknown. The project used application-generated citations from JSON; it should not be read as a demonstration of Anthropic’s later native citations API, whose launch is recorded in the Claude Platform release notes.
The resulting system extracted company details and customer insights with a verifiable trail back to the original transcripts. That trail made it possible to inspect the basis of a finding instead of accepting the model’s answer on authority. But richer analysis and the effort to keep errors low increased the cost substantially.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce repeated input work and split outputs
The workflow often reached the 4,000-token output limit Guo encountered with Claude 3.5 Sonnet. Completing one transcript’s analysis could therefore require several requests. Two features that were experimental at the time addressed different sources of repeated work.
Prompt caching addressed repeated input. The team reused the same transcript to extract metadata and then insights, so caching let later requests reuse that transcript content. Guo cites reductions of up to 90% in cost and up to 85% in latency for prompt caching. Those figures match the ceilings advertised in Anthropic’s prompt-caching announcement, rather than a supplied measurement of this project’s overall performance. Cache writes carry a premium, and the announcement’s latency example measures time to first token; the benefit depends on reusing the long prompt.
Extended outputs addressed incomplete responses. An experimental feature flag doubled the available output capacity, allowing complete summaries in a single pass instead of splitting the analysis across multiple turns. This expanded the amount the model could generate, not its input context window. Historically, Anthropic introduced the 8,192-token output beta in July 2024 and removed the beta-header requirement in August; the flag described here belongs to that earlier API period.
Guo reports that the optimizations reduced the analysis cost from $5,000 to $500 and delivered results in days instead of weeks. That is his project-level account, without a billing or runtime breakdown. The engineering mechanisms are distinct: caching reduces repeated processing of the transcript, while longer outputs reduce the need for additional requests to finish a summary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The research infrastructure reaches other teams
The larger surprise was how widely the work became useful. The initial customer was the executive team, which wanted insights that Guo says would normally require tens, if not hundreds, of hours to produce. Once the underlying transcript workflow existed, other teams could use it for their own work.
- Marketing: Retrieved customer quotes for branding and positioning exercises.
- Sales: Automated transcript downloads using the system. Guo reports that automating transcript downloads saved the sales team dozens of hours every week.
The value extended beyond the generated research: even the surrounding data-access machinery removed recurring work.
Teams also began asking questions they previously would not have considered because manual analysis was too daunting. Making the corpus accessible changed which research questions were practical to pursue, turning an accumulation of unstructured conversations into a shared source of evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the system around the model
Model capability mattered: in Guo’s experiments, Claude 3.5 Sonnet and GPT-4o handled tasks the other tested models could not. But the final choice combined accuracy with prompt caching. Model fit depended on both the errors the project could tolerate and the economics of repeatedly analyzing long transcripts.
The surrounding software mattered just as much. Structured JSON, good database schemas, and the overall architecture produced significant gains. AI engineering means building an effective system around the model: its outputs need to fit the application’s data structures and workflows, and its evidence needs to remain available to the people using the results. Guo’s lesson is to integrate the model thoughtfully into the system rather than add it as an afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make customer research a repeatable operation
The project did not end with a report or white paper. The team built an interface around the analysis with search, filters, and exports. Those features let colleagues find relevant material and carry it into their own work, making a simple, flexible tool useful across the company.
The purpose was to augment human analysis by removing the bottlenecks that prevented it. Once downloading, reading, and organizing the corpus no longer consumed the entire research budget, people could investigate questions that had previously been out of reach. Faster processing enabled a broader range of work.
That leaves a practical question for any organization: what customer data is already available but rarely examined? Sales calls, support tickets, product reviews, user feedback, and social media interactions all contain potential evidence. Guo’s closing challenge is to use the available tools and techniques to make those neglected sources accessible—and begin turning the accumulated data into something people can use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Charlie Guo's earlier account of the sales-call analysis project, including model selection, transcript caching, citations and the interface built for colleagues.
- Prompt caching with ClaudeArticle
Anthropic explains prompt reuse, cache pricing and example cost and time-to-first-token improvements for long inputs.
- Claude Platform release notesDocumentation
Dated API history covering Claude 3.5 Sonnet's longer outputs, prompt caching and subsequent citation support.
Full transcript
Loading transcript…