# Building Alice’s Brain: an AI Sales Rep that Learns Like a Human - Sherwood & Satwik, 11x

Sherwood Callaway · Satwik Singh

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/KWmkMV0FNwQ-building-alices-brain-ai-sales-rep-that) · [Video](https://www.youtube.com/watch?v=KWmkMV0FNwQ)

## Building Alice’s Brain: an AI Sales Rep that Learns Like a Human

Alice’s knowledge base turns seller documents, websites and media into retrievable context, then makes the evidence behind sales outreach visible to users.

Familiarity with language-model context windows, embeddings and basic web application architecture will help; sales terminology is introduced as needed.

### What does Alice need to write a useful sales email?

[0:25](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=25s)

A sales email needs more than a recipient’s name. It needs a reason that this seller’s offer matters to this particular person. That is the problem behind Alice, 11x’s AI sales development representative, or **SDR**. Sherwood Callaway, who introduces himself as the engineering lead for Alice, presents the system with Satwik Singh. Alongside Alice, 11x also builds Julian, a voice agent. An SDR sources leads, engages them across channels and books meetings; the outcomes that matter are positive replies and meetings booked. The opening example is an email Alice has written—the everyday output that her knowledge base must support.

Sherwood reports that Alice sends about 50,000 emails per day, compared with 20–50 for a human SDR. At the time of the talk, Alice was running campaigns for about 300 organizations. These are reported operating volumes, not measurements of reply rates or meetings booked; the human comparison’s conditions are not specified.

![Email interface showing the subject “Quick thought about AudioMob’s financials” and a complete outreach message addressed to Norbert.](https://ai.engineer/api/talk-essay-media/ddc6ad9ee11bb330976175f21b800ecf2fa647481248f88b1327956f88f7d712)

An example sales email referencing AudioMob’s growth and expense management.

Two terms separate the parties involved: the **seller** is the company using Alice to sell something, and the **lead** is the person receiving the outreach. The seller supplies business context, such as products and case studies, which Alice uses to personalize emails.

Alice needs knowledge on both sides of that relationship:

- **Seller context:** Products, services, case studies, pain points, value propositions and the ideal customer profile, or ICP.
- **Lead context:** Role, responsibilities, priorities, previously tried solutions, pain points and employer.

The knowledge-base work here focuses on knowing the seller: making its business information available when Alice needs to write.

### The limits of a manually curated library

[3:09](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=189s)

Originally, sellers pushed context into Alice through a feature called the library. They entered each product and service, then filled in descriptions of its pain points, solutions and value propositions. These were not merely dashboard labels: the descriptions became context for email generation. During campaign creation, users selected which library offers Alice could access.

That workflow created two connected problems. First, onboarding required substantial manual entry, and users could not run campaigns until they had populated their library. Second, selecting offers forced a poor tradeoff: too few could leave Alice with nothing relevant for a particular lead, while too many filled the context window with material that made the resulting emails worse.

**The change was to move context selection from the seller to Alice.** Instead of asking users to describe and select every useful offer, the product would let Alice pull seller information into a knowledge base and retrieve the relevant material when writing each email.

### From uploaded resources to searchable knowledge

[5:35](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=335s)

The new onboarding model resembles training a human SDR: provide source documents, allow time to learn and check what the person understands. For Alice, the equivalent is a centralized repository of seller information that can be consulted during message generation. Marketing materials, case studies, sales calls and press releases enter through three processing groups: documents and images, websites, and audio and video.

The ingestion architecture separates accepting a resource from making it searchable:

1. **Accept the upload.** The client uploads a resource, which is saved to S3 and passed to the backend.
2. **Record and dispatch it.** The backend creates database records and starts jobs according to the resource type and selected parsing vendor.
3. **Wait for parsing.** Vendors process resources asynchronously, then send completion webhooks to the ingestion path.
4. **Persist and index the result.** The application stores the parsed artifact in its database and embeds and upserts its content into Pinecone.
5. **Expose the knowledge.** Database updates refresh the UI, while the agent can query Pinecone for the newly stored information.

The final architecture reveal connects the upload and processing infrastructure to that agent query.

![Architecture diagram connecting a client, Amazon S3, GQL server, internal server, parsing API, Inngest, PostgreSQL, webhook handler, event consumer and Pinecone; the agent and query connection are highlighted.](https://ai.engineer/api/talk-essay-media/eac8994757cdc4cbc23ec017e32e0c5d82abefc579a66a610abc57275357898f)

Resource ingestion architecture, ending with an agent querying Pinecone.

Within this architecture, the team identifies five pipeline stages: **parsing, chunking, storage, retrieval and visualization**. The last stage is part of the product contract: users need a way to inspect what Alice has learned, not just a background process that finishes successfully.

### Make every resource usable as Markdown

[7:52](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=472s)

Parsing converts resources such as PDFs, images and MP4 files into text that the downstream language-model workflow can use. Multimodal models offer another way to consume some of these inputs, but their restrictions did not eliminate the need for parsing in this system. Alice’s common output format is **Markdown**, which preserves meaningful distinctions such as headings, paragraphs and formatting rather than flattening everything into undifferentiated text.

The team chose not to build parsers from scratch. Supporting five resource types meant supporting many individual file formats, and each format could demand specialist knowledge. Vendors already concentrated their engineering effort on those problems. Outsourcing parsing offered a faster route to market and a more credible outcome than asking the Alice team to become experts in every input format.

Initial vendor selection used three requirements: support the necessary resource types, return Markdown and deliver results through webhooks. Two quality dimensions were deliberately not benchmarked at first:

- **Accuracy:** Whether the extracted output matches the original resource.
- **Comprehensiveness:** How much of the resource’s information appears in the output.

The team assumed leading vendors would fall within a reasonable band on both. Cost optimization was also deferred: the system was still pre-production, so they lacked the usage data needed to understand where spending would accumulate.

### Different parsers, one downstream format

[10:56](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=656s)

For documents and images, the team selected [LlamaParse](https://github.com/run-llama/llama_cloud_services), a LlamaIndex product. Among the document parsers they evaluated, it offered the broadest file-format support they found. Support also mattered: the team describes Jerry and the team joining a Slack channel within a couple of hours of the initial evaluation. The demonstration converts an 11x sales-deck PDF into Markdown, turning presentation content into material the rest of the pipeline can process.

For websites, they selected [Firecrawl](https://docs.firecrawl.dev/features/crawl). They already knew it from another project, and the crawl endpoint they needed from the alternative, [Tavily](https://docs.tavily.com/documentation/api-reference/endpoint/crawl), was still in development during selection. That was an availability decision, not a finding that Tavily parsed websites poorly; Tavily now documents a crawl endpoint. The example takes the Brex homepage and produces another Markdown document.

For audio and video, they chose [Cloudglue](https://docs.cloudglue.dev/getting-started/video-to-text). The decisive capability was extracting information from the video imagery as well as transcribing speech. A transcript alone can miss information presented visually. The talk describes YouTube videos, MP4 files and other video formats becoming Markdown. Input route and API still matter when reproducing that behavior today: Cloudglue’s current Describe documentation demonstrates uploaded-video visual extraction, while its current [Extract guide](https://docs.cloudglue.dev/deep-dives/extraction-guide) limits direct YouTube entity extraction to speech. The talk does not specify Alice’s historical YouTube processing route.

### Preserve structure before enforcing size

[12:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=759s)

Once every resource is Markdown, chunking turns long documents into units that can be embedded and retrieved. A title and a paragraph carry different kinds of meaning, so preserving structure matters. Retrieving an entire ten-page document would also make it harder to isolate the part relevant to a particular email.

Possible boundaries include tokens, sentences and Markdown headers; an LLM can also decide how to split a document. The choice starts with the intended retrieval unit: what information should stay together, what will the agent ask for, and will the same approach work for a PDF, a slide deck and a video-derived document? A chunk must preserve enough meaning to be useful while remaining suitable for embedding and storage.

Alice uses a **waterfall**: split on Markdown headers first, then sentences, then tokens as needed to satisfy a target token count. The target size is not specified in the talk. The following TypeScript expresses that control flow, with the boundary splitters and tokenizer supplied by the implementation:

```typescript
type Split = (text: string) => string[];

type ChunkingRules = {
  maxTokens: number;
  countTokens: (text: string) => number;
  byMarkdownHeader: Split;
  bySentence: Split;
  byTokenLimit: Split;
};

function chunkMarkdown(text: string, rules: ChunkingRules): string[] {
  const splitters = [
    rules.byMarkdownHeader,
    rules.bySentence,
    rules.byTokenLimit,
  ];

  function split(part: string, level: number): string[] {
    if (rules.countTokens(part) <= rules.maxTokens) return [part];
    if (level === splitters.length) {
      throw new Error("Token splitter exceeded the configured limit");
    }
    return splitters[level](part).flatMap(piece => split(piece, level + 1));
  }

  return rules.byMarkdownHeader(text).flatMap(part => split(part, 1));
}
```

A section that already fits stays intact. An oversized section falls through to sentence boundaries, and an oversized sentence reaches the token splitter. This makes size enforcement progressively more aggressive instead of discarding structural boundaries immediately.

The team reports that the approach worked across their document types and preserved Markdown chunks they could display cleanly in the UI. It also prevented very long chunks from diluting the meaning of the information being embedded. That connects chunking to two consumers at once: retrieval needs focused units, and users need readable evidence.

### Choose storage for the retrieval operation

[14:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=898s)

**Retrieval-augmented generation does not require a vector database.** External context can come from a graph database, Elasticsearch, a document or relational database, a key-value store, files or S3. Alice uses vector storage because the operation the team wanted was similarity search.

The team chose Pinecone as an established, cloud-hosted provider with approachable guides and SDKs. Hosting removed another infrastructure responsibility. Its bundled embedding models also removed the need to select a separate embedding provider or host a model themselves: they could use the embedding capability supplied with the database service. Vendor support contributed beyond setup, helping the team examine alternative storage approaches, including graph databases and graph RAG.

### Let the lead guide the research

[16:54](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1014s)

With the seller’s information indexed, the next question is how to retrieve it. The talk describes a progression in which retrieval becomes increasingly controlled by the agent:

| Approach | How retrieval enters generation |
| --- | --- |
| Traditional RAG | Retrieved context enriches the system prompt. |
| Agentic RAG | The agent calls retrieval tools within its workflow. |
| Deep research RAG | The agent plans retrieval and decides whether to search further. |

In the last approach, research can broaden or deepen according to what the task still needs. Retrieval is no longer necessarily a single fixed step before generation.

Alice’s team built its research agent with [Letta](https://docs.letta.com/). The workflow begins with **lead information**, which conditions the research plan:

1. Form a plan containing one or more context-retrieval steps.
2. Execute the retrieval tool calls.
3. Summarize the results.
4. Return answers in a clean Q&A format.

The demonstration shows a result organized around two questions. The important connection is between lead-specific research and seller knowledge: the repository can contain broad business context, while the agent selects what is useful for the current recipient.

### Make Alice’s knowledge inspectable

[18:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1110s)

A seller is trusting Alice to represent its business. That creates concrete concerns: will she describe the products correctly, invent a case study or testimonial, or claim to solve a pain point the business does not address? The team’s response was to let users inspect the knowledge behind the output.

The product projects vectors from Pinecone into three dimensions using [UMAP](https://arxiv.org/abs/1802.03426), then renders them as interactive nodes. Clicking a node reveals its associated text chunk. Sales and support teams use this view to demonstrate what Alice knows. The useful inspection operation is the connection from a visible node to stored source text; the spatial display itself is not a correctness test.

![Product screenshot showing a network of colored nodes and connecting lines beside a text panel; Visualization is highlighted in the pipeline sidebar.](https://ai.engineer/api/talk-essay-media/022a35aac47a1c9759cda7ad313db3107b7dda9f9dc32fd82be74aa2ac6b6c97)

UMAP projection of the knowledge-base vectors, with a selected text passage alongside. The interface labels this view “Knowledge Graph Visualization.”

The ordinary onboarding path begins with an upload modal. Users add URLs, web pages, documents or videos and click **Learn**; the resources then appear in the knowledge-base UI. A separate Letta-powered agent lets users ask questions about that knowledge.

During campaign creation, knowledge-base content appears as Q&A on the left side of the interface. Expanding a question reveals the retrieved chunks used in the messaging flow. This brings evidence inspection into the task where it matters: a user can examine the context used to produce outreach instead of manually choosing every offer beforehand. The resulting experience returns to the human SDR analogy—provide the source material, then inspect how it is being used.

### Reach production, then measure the remaining problems

[20:38](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1238s)

The difficulty of this project came from the accumulation of small decisions: choosing technologies, supporting heterogeneous resource types and connecting their outputs into one usable workflow. Sherwood describes RAG as substantially harder than the team expected. Their production advice follows from that experience: first ship a system that satisfies the product requirements, then establish benchmarks from real usage and iterate.

Vendors were part of that learning process, not just interchangeable endpoints. The team recommends asking vendors to explain their offerings, compare alternatives and help customers understand why a particular approach fits. Those conversations can reduce the amount of specialist knowledge a small product team must acquire alone.

The remaining work was explicitly a future agenda:

- **Email hallucinations:** Track and address unsupported claims in generated outreach.
- **Parsing quality:** Evaluate vendors on accuracy and completeness, the dimensions deferred during initial selection.
- **Hybrid retrieval:** Experiment with a graph database alongside vector storage.
- **Pipeline cost:** Reduce spending across the full ingestion and retrieval path.

The implemented knowledge base changed how sellers supplied and inspected context. The next phase was to measure whether the extracted information was faithful and complete, whether Alice used it reliably, and what that reliability cost.

![Future plans slide listing tracking email hallucinations, evaluating parsing vendors, experimenting with vector and graph databases, and reducing pipeline cost.](https://ai.engineer/api/talk-essay-media/cdba98788dfce268b098447f4c44ab5c983451ceb567e8d5fbf104f8e591c9d2)

Future plans cover hallucinations, parsing quality, hybrid RAG and pipeline cost.

## Resources

- [Firecrawl crawling documentation](https://docs.firecrawl.dev/features/crawl) — Crawl websites into Markdown and collect results through polling or webhooks. Examples use the current API.
- [Tavily Crawl documentation](https://docs.tavily.com/documentation/api-reference/endpoint/crawl) — The currently documented crawl endpoint supports website traversal and Markdown output.
- [Cloudglue video-to-text guide](https://docs.cloudglue.dev/getting-started/video-to-text) — Convert uploaded video into Markdown containing speech, visual descriptions and on-screen text.
- [Cloudglue structured extraction guide](https://docs.cloudglue.dev/deep-dives/extraction-guide) — Extract structured entities from media, with guidance on direct YouTube inputs and uploaded files.
- [Letta documentation](https://docs.letta.com/) — Current guides for building stateful agents and embedding them in applications.
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://arxiv.org/abs/1802.03426) — The research paper introducing the dimensionality-reduction method used for Alice's three-dimensional knowledge view.

## Slides

### Innovation partner and platinum sponsors

AWS is the innovation partner, with six companies listed as platinum sponsors.

[0:10](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=10s) [AWS innovation partner logo above six platinum sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/95b686ba8a0c0f9201888395a8b899b32d3d8afda1fb8489a11b50d452ba5c42.webp)

INNOVATION PARTNER
aws

PLATINUM SPONSORS
Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Speakers

Sherwood Callaway leads engineering for 11x’s Alice product; Satwik Singh is the AI lead.

[0:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=23s) [Speaker cards for Sherwood Callaway, Tech Lead, Alice, and Satwik Singh, AI Lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/8104ea0c9be6ed2fd3cd6ba6664688581f81b85a3d2ec8097d0b12e9d0c06292.webp)

Speakers

Sherwood Callaway
Tech Lead, Alice

Satwik Singh
AI Lead

Interface text:
AIE
Microsoft
smol ai

### What is an SDR?

An SDR is an entry-level sales representative who sources leads, contacts them across channels, and books meetings.

[0:56](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=56s) [An SDR is a sales development representative.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3069a1aae5ec441742e21e944aaf8a83750e981c02d1081eee79c20ce862173.webp)

What is an SDR?

Sales Development Representative

AIE
aws

### What is an SDR?

An SDR is a sales development representative, an entry-level sales role focused on finding and contacting leads and booking meetings.

[1:03](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=63s) [An SDR is a sales development representative.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb92b730d933dce20fd811f735910d4cbce49826cd2fdaa5eb312e658370fc87.webp)

What is an SDR?
Sales Development Representative

AIE
aws

### What is an SDR?

An SDR sources prospective customers, contacts them across channels, and books meetings. The role is measured by positive replies and meetings booked.

[1:06](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=66s) [An SDR is an entry-level sales role with responsibility for sourcing leads.](https://assets.ai.engineer/sites/aiecode2025/sha256/fff87796b23f2ab429b608d91a8b2decc23a8a9cec75d6e108530c5df4d54cac.webp)

What is an SDR?

Sales Development Representative

Entry-level sales role

Responsibilities:
• Sourcing leads

Interface text:
AIE
Microsoft
smol ai

[1:27](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=87s) [An SDR sources leads, contacts them and books meetings; key metrics are positive replies and meetings booked.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba998ddf6765cc4e77970ea9250f714c0c1720d757e55559e85dcb65a4ba7572.webp)

What is an SDR?

Sales Development Representative

Entry-level sales role

Responsibilities:
• Sourcing leads
• Contacting them
• Booking meetings

KPIs: Positive Replies, Meetings Booked

Interface text:
AIE
aws

### Sales email example

Alice’s example email ties AudioMob’s growth and partnerships to expense administration, then offers a link. The speaker says Alice sends about 50,000 such emails a day, versus 20–50 for a human SDR.

[1:32](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=92s) [A sales outreach email references a company’s growth and expense-management challenges and offers to share a link.](https://assets.ai.engineer/sites/aiecode2025/sha256/515ac286d5763774dce09687d63e2e5f8fc577c73c42f4954c4514cbf9c97fda.webp)

Subject line: Quick thought about AudioMob's financials

Hey Norbert,

Fantastic work on AudioMob's 150% revenue growth this year! With that rapid scaling, I imagine keeping track of your team's expenses is becoming more complex - especially with 200+ game developer partnerships to manage.

We've helped other fast-growing tech companies cut their expense admin time by over 80%, giving their teams more time to focus on innovation rather than chasing receipts.

If you're up for it, I can send over a quick link to check it out - no pressure, just when you're ready.

Want me to?

Interface text:
Norbert Nyamikeh
Financial Controller at AudioMob
Sequence
Research
Step 1
Email sent
24 days ago
1
AIE
Microsoft
smol ai

### Message volume over time

Sent and inbound message counts both rise from January to April, reaching 2,632,699 and 20,671 respectively on April 21. The chart uses separate vertical scales.

[1:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=118s) [Line chart of sent and inbound messages through April 2025.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec6200d419e789abf46df8ada5d79bf48a44bf977634a022a3d4dde9a2d2587a.webp)

21 Apr 2025 (UTC)
A  sent_messages  2,632,699
B  inbound_messages  20,671

Left Y-axis ticks: 0, 500,000, 1,000,000, 1,500,000, 2,000,000, 2,500,000, 3,000,000
Right Y-axis ticks: 0, 5,000, 10,000, 15,000, 20,000, 25,000
Timeline: 1-Jan-2025, 4-Jan-2025, 7-Jan-2025, 10-Jan-2025, 13-Jan-2025, 16-Jan-2025, 19-Jan-2025, 22-Jan-2025, 25-Jan-2025, 28-Jan-2025, 31-Jan-2025, 3-Feb-2025, 6-Feb-2025, 9-Feb-2025, 12-Feb-2025, 15-Feb-2025, 18-Feb-2025, 21-Feb-2025, 24-Feb-2025, 27-Feb-2025, 2-Mar-2025, 5-Mar-2025, 8-Mar-2025, 11-Mar-2025, 14-Mar-2025, 17-Mar-2025, 20-Mar-2025, 23-Mar-2025, 26-Mar-2025, 29-Mar-2025, 1-Apr-2025, 4-Apr-2025, 7-Apr-2025, 10-Apr-2025, 13-Apr-2025, 16-Apr-2025, 19-Apr-2025

Interface text:
AIE
Microsoft
smol ai

### Defining terms

The seller is 11x’s customer: a company selling through Alice. The lead is the person that company is selling to.

[2:02](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=122s) [Defines seller as the company selling through Alice and lead as the person being sold to.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff7fc71c9926c3d3c25dd63d13002d083de6f4de97095b16af267287c440c94e.webp)

Defining terms

Seller
The company that is selling something (through Alice)

Lead
The person being sold to

Interface text:
AIE
aws

### Seller, Alice and leads

The seller gives Alice business context, including products and case studies. Alice uses it to personalize emails to individual leads.

[2:22](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=142s) [Diagram shows a seller providing context to Alice, who sends emails to three leads.](https://assets.ai.engineer/sites/aiecode2025/sha256/377d12bd3dd0a3b0a838593e6417701365c47dd0f7a43fe478905d4fb4c9a79d.webp)

Seller

Alice the SDR
[illegible]

Lead
Lead
Lead

AIE
aws

### Requirements for Alice to succeed

Alice needs to know the seller’s offerings and the lead’s role, company, and pain points to personalize outreach.

[2:40](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=160s) [Two requirements: know the seller and know the lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/7afa8332682bfb40b229bb0afedc9162708e79e0c32f4a47ac941f2170470e61.webp)

Requirements for Alice to succeed

Know the Seller
Know the Lead

AIE
aws

[2:49](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=169s) [Two requirements: know the seller and know the lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/c18752ddb4cb4be63a9844c4187946593d373777da256dfe3d778f3f772692ba.webp)

Requirements for Alice to succeed

Know the Seller
Know the Lead

AIE
Microsoft
smol ai

### Knowing the seller

Alice needs to know both the seller and the lead. The talk focuses on seller context, including products, services, case studies, pain points, and value propositions.

[3:04](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=184s) [The requirements slide highlights Know the Seller beside Know the Lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/401165498510059a22cc3ce264cfe8b2ae8e7596b1ba3cfbc9d77b855994334a.webp)

Requirements for Alice to succeed

Know the Seller
Know the Lead

AIE
Microsoft
smol ai

### Seller context and lead outreach

In the old product, sellers manually supplied business context through a library. Alice could reference that context when writing emails to leads.

[3:07](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=187s) [A seller provides context to Alice, who sends messages to three leads.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f18c249a93747f028df8d81339866f16f48c3a485bd4716e23467fe2f5d1c6c.webp)

Seller

Alice the SDR
[illegible]

Lead
Lead
Lead

Interface text:
AIE
aws

### Pleo for Startups offer editor

In the old product, sellers manually entered offer details for Alice. This example organizes Pleo’s offer into pain points, solutions and proof points; their descriptions became context for her emails.

[3:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=210s) [An offer editor displays columns for pain points, solutions and proof points.](https://assets.ai.engineer/sites/aiecode2025/sha256/bdf3343474a58c5a9dfd35519dbc3afc49f71a2bd7899678182151378fe8ccd7.webp)

Pleo for Startups
Pleo is a business spend management solution designed to simplify expense reporting and empower teams to spend autonomously. It offers smart company cards, automated expense management, and real-time spending overviews, making it ideal for startups looking to streamline their financial processes.

Pain points
Time-consuming expense management
Lack of spending oversight
Complex reimbursement processes
Name *
Enter name
Description *
Enter description
Add
Connect existing pain points
Search for pain points...

Solutions
Smart Company Cards
Integrated Accounting System
Real-time Spending Overview
Automated Expense Management
Name *
Enter name
Description *
Enter description
Add
Connect existing solutions
Search for solutions...

Proof points
Increased Employee Autonomy
Time Savings on Admin Work
Improved Financial Oversight
Name *
Enter name
Description *
Enter description
Add
Connect existing proof points
Search for proof points...

Interface text:
11x
Search
Create New Campaign
Campaigns
Hot Leads
Output
Library
Analytics
Settings
Log Out
Cancel
Save
AIE
aws

[3:45](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=225s) [An offer editor organizes pain points, solutions, and proof points in three columns.](https://assets.ai.engineer/sites/aiecode2025/sha256/cee67b6ba9a4f71877d6bd447f4d96d54ff0908e2018232f095494b28bd352b9.webp)

Pleo for Startups
Pleo is a business spend management solution designed to simplify expense reporting and empower teams to spend autonomously. It offers smart company cards, automated expense management, and real-time spending overviews, making it ideal for startups looking to streamline their financial processes.

Pain points
Time-consuming expense management
Lack of spending oversight
Complex reimbursement processes
Name *
Enter name
Description *
Enter description
Add
Connect existing pain points
Search for pain points...

Solutions
Smart Company Cards
Integrated Accounting System
Real-time Spending Overview
Automated Expense Management
Name *
Enter name
Description *
Enter description
Add
Connect existing solutions
Search for solutions...

Proof points
Increased Employee Autonomy
Time Savings on Admin Work
Improved Financial Oversight
Name *
Enter name
Description *
Enter description
Add
Connect existing proof points
Search for proof points...

Interface text:
11x
Search
Create New Campaign
Campaigns
Hot Leads
Output
Library
Analytics
Settings
Log Out
Cancel
Save
AIE
Microsoft
smol ai

### Campaign messaging setup

During campaign setup, the user selects library offers for an example lead. Alice can use those offers when generating emails.

[3:50](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=230s) [A campaign setup screen shows an example lead, selectable offers, and an email being generated.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1051d4b8e636185d1ac12521a0d97e5213efeb03060fcc9e374e99dccd75cba.webp)

New Campaign

Sequence Example Lead
These details will be used as an example for the email sequences.
Karen Ehrenberg
Customer Service Manager at the Pencil Grip, Inc.
Manager
Sierra Madre, California, United States
Alice is researching...

Select offers from your library:
Cashback 1 Pager
Cashback Trigger
CFO Playbook 2025
Customer Stories Test
Dutch Offer
FR Offer
Integrations
Lead Qualification
NEAT-Qualification Framework
Objection Handling
Overdraft
PayG to Contract Trigger - Essential Plan
Pleo for Agencies
Pleo for Healthcare
Pleo for Larger businesses
Pleo for Larger businesses FR
Pleo for Larger businesses NL
Pleo for Retail
Pleo for Small businesses
Pleo for Startups
Pleo for Technology
Pleo for Transport
Pleo Pricing
Pleo x TravelPerk
Reimbursements
Security & SAML SSO
Spend Controls
Vendor Cards
Create new offer

Edit campaign language: English
Scraping the Web 45%
Alice AI
Template Mode
Step 1
Email
Generating Message...
Subject:

Interface text:
Audience > Messaging > Settings
Type to edit messaging
Save as Draft
Next
AIE

### Library problems

Filling out the offer library was tedious: users had to enter substantial information before they could run campaigns.

[4:07](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=247s) [A slide titled Library problems lists tedious work.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6c63847132aef42a9dbb48ab17bb79b1eab821aa364c16ab6bb6d73b7774364.webp)

Library problems

• Tedious

Interface text:
AIE
aws

### Library problems

Filling the library was tedious and blocked campaigns. Too few offers risked irrelevance to a lead; too many filled Alice’s context window and weakened her emails.

[4:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=279s) [A slide lists tedious work, onboarding friction, and suboptimal emails as library problems.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc2c48f37955a47368847a41df406386710737434fa27d8f6b2b67cb45766e95.webp)

Library problems

• Tedious
• Onboarding friction
• Suboptimal emails

Interface text:
AIE
aws

### Seller context and lead outreach

Alice reverses the flow of context: instead of relying on the seller to push business information, she pulls it into a knowledge base and selects what is relevant when writing emails to leads.

[4:47](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=287s) [A diagram connects a seller to an AI agent and the agent to three leads by email.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1c6e76ea07475ab489b140e4eaa3c202c773c93c163bf4e3cb48d53eddf4470.webp)

Seller
Alice the SDR
[illegible]
Lead
Lead
Lead

Interface text:
AIE
Microsoft
smol ai

[4:56](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=296s) [Diagram showing an AI sales agent connected to a seller and three leads by arrows and email icons.](https://assets.ai.engineer/sites/aiecode2025/sha256/095c68df393f18d665743c670645432541a67794b42bbceb71833460f7846774.webp)

Seller
Alice the SDR
[illegible]
Lead
Lead
Lead

Interface text:
AIE
aws

### Agenda

The talk covers the knowledge base, its RAG pipeline and user experience, then closes with lessons and future plans.

[5:28](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=328s) [Agenda lists a high-level overview, pipeline deep-dive, user experience, and lessons and future plans.](https://assets.ai.engineer/sites/aiecode2025/sha256/4bd1917b67f8f83b4b44d9ce7cb202c342ab5f06273682c3b9f09e208d160c8c.webp)

Agenda
1. High-level overview
2. Deep-dive on pipeline
3. User experience
4. Lessons & future plans

AIE
Microsoft
smol ai

### Agenda

The talk moves from a knowledge base overview to the RAG pipeline and user experience, then closes with lessons and future plans.

[5:31](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=331s) [Agenda lists a high-level overview, pipeline deep-dive, user experience, and lessons and future plans.](https://assets.ai.engineer/sites/aiecode2025/sha256/d42d23975a80a31ccf650aadc53f904cd234be37cac77715da7228b6837748b7.webp)

Agenda
1. High-level overview
2. Deep-dive on pipeline
3. User experience
4. Lessons & future plans

AIE
Microsoft
smol ai

### Overview

The overview begins with a comparison to training a human sales development representative on documents.

[5:32](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=332s) [Overview section title over a desert landscape.](https://assets.ai.engineer/sites/aiecode2025/sha256/e59edaee7dab48b48465b54f5a0722304c909c76515e7ca9789ec325681dece2.webp)

Overview

11x
AIE
Microsoft
smol ai

### Introducing the Knowledge Base

Users add seller information from sources such as PDFs, websites, and call recordings. Alice highlights the most relevant information for each lead, and the platform can reference it when generating messages.

[5:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=339s) [Slide describes the knowledge base as a centralized repository for seller information.](https://assets.ai.engineer/sites/aiecode2025/sha256/bbb41f8e6a1060cd96e173c8d2aacf5b462588ede5e3ea9cf49c43f8cf06aac5.webp)

Introducing the Knowledge Base

Centralized repository for seller info

Interface text:
AIE
aws

[6:08](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=368s) [The knowledge base centralizes seller information from multiple sources, and Alice highlights relevant information for each lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/eae13f7fee4bde5dfeb554b1c093eaf4b85714711be5ee84ba4660c4971a9597.webp)

Introducing the Knowledge Base

Centralized repository for seller info

Users can add info from any source (PDFs, websites, call recordings, etc.)

Alice highlights the most relevant info for each lead

Interface text:
AIE
aws

### Documents and images

11x chose LlamaParse for document and image parsing because it supported many file types and the team received quick customer support.

[11:14](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=674s) [LlamaIndex is shown for document and image parsing, with file-type support and customer support listed as reasons.](https://assets.ai.engineer/sites/aiecode2025/sha256/709a994074b45c047d8f88a7e65eb548dcd537dfef837657745779528852c535.webp)

Documents & images

LlamaIndex

Reasons:
• Supports a huge # of file types
• Good customer support

Parsing
↓
Chunking
↓
Storage
↓
Retrieval
↓
Visualization

AIE

### Document parsing example

LlamaParse turns an 11x sales deck PDF into Markdown, extracting headings, descriptions and key statistics from the document.

[11:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=683s) [A sales deck PDF is shown alongside its extracted Markdown.](https://assets.ai.engineer/sites/aiecode2025/sha256/c039aec0008f2913b323b4ca215a09c72b50c91623cae918c186fa2e4473d69e.webp)

Parsing
↓
Chunking
↓
Storage
↓
Retrieval
↓
Visualization

PDF:
11x
The Home of AI Digital Workers
Trusted by the best
pwc  gupshup  MERCK  sumup  GoStudent  PLEO  Rho  you.com  YouGov
We build your Digital Workforce
200 Years

Extracted Markdown:
# 11x
# The Home of AI Digital Workers

Trusted by the best
pwc gupshup MERCK sumup GoStudent PLEO Rho you.com YouGov

Alice the SDR
Autopilot activated

Alice the SDR
Hey James, have you tried hiring an AI SDR?
---
# We build your Digital Workforce

The future of work is digital. Our workers enable teams to focus on impactful work, driving growth and innovation.

## AI Workers

### Alice — AI SDR

Alice drives email and LinkedIn outreach, mastering your playbook to convert prospects into high-value leads.

### Mike — AI Phone Agent

Mike is the ultimate phone rep, learning from every call, adapting to your business, and building relationships 24/7.

## Key Statistics

200 Years
Freed from repetitive tasks

1000s
Digital AI workers deployed globally

$70M+
andreessen horowitz  HubSpot Ventures  BENCHMARK

...

Interface text:
11x Sales Deck - Generic.pdf
Page 1 of 29
AIE
Microsoft
smol ai

### Websites

11x chose Firecrawl for website parsing because the team knew it from a previous project and Tavily’s /crawl endpoint was still in development at the time.

[11:45](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=705s) [Firecrawl is shown for website parsing, with familiarity and the then-unavailable alternative crawl endpoint listed as reasons.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9ee42866c4dd879ba1593e00d21ef83335edfc472b7b7e89fbae52d72af0d25.webp)

Websites

🔥 Firecrawl

Reasons:
• Familiar
• Tavily /crawl endpoint still in development

Parsing → Chunking → Storage → Retrieval → Visualization

AIE
Microsoft
smol ai

### Website parsing example

Firecrawl turns the Brex homepage into a Markdown document, extracting its headings, copy and image references for the parsing workflow.

[11:55](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=715s) [A Brex homepage is shown alongside extracted Markdown.](https://assets.ai.engineer/sites/aiecode2025/sha256/f71de3aeff07b28b2303299fe69a8f358cd52a5a4392625085cad6ebaf11012b.webp)

Parsing
Chunking
Storage
Retrieval
Visualization

Brex homepage:
Spend smarter. Move faster.
Gain control and speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.
Get started
See Brex in action

Extracted Markdown:
[Meet Brex for Zip! Faster procurement and payments in one workflow →](https://www.brex.com/brex-for-zip-procurement)

![HP Hero Desktop](https://brand.brex.com/transform/[illegible])![HP Hero Mobile-New](https://brand.brex.com/transform/[illegible])

# Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

Brex is a financial technology company, not a bank. Checking and banking services provided by Column N.A., Member FDIC.

See Brex in action

![HP Hero Desktop](https://brand.brex.com/transform/[illegible])![HP Hero Mobile-New](https://brand.brex.com/transform/[illegible])

# Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

Brex is a financial technology company, not a bank. Checking and banking services provided by Column N.A., Member FDIC.

See Brex in action

## Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

See Brex in action

...

Interface text:
Brex
Products
Solutions
Resources
Customers
Pricing
Sign in
See a demo
Get started
Enter your work email
AIE
Microsoft
smol ai

[11:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=718s) [A website is converted into a Markdown document in the parsing workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd86b087333c11d627ac6de75c160148d278daeb02a65492e6060c7444a168f1.webp)

Parsing
Chunking
Storage
Retrieval
Visualization

Brex homepage:
Spend smarter. Move faster.
Gain control and speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.
Get started
See Brex in action

Extracted Markdown:
[Meet Brex for Zip! Faster procurement and payments in one workflow →](https://www.brex.com/brex-for-zip-procurement)

![HP Hero Desktop](https://brand.brex.com/transform/[illegible])![HP Hero Mobile-New](https://brand.brex.com/transform/[illegible])

# Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

Brex is a financial technology company, not a bank. Checking and banking services provided by Column N.A., Member FDIC.

See Brex in action

![HP Hero Desktop](https://brand.brex.com/transform/[illegible])![HP Hero Mobile-New](https://brand.brex.com/transform/[illegible])

# Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

Brex is a financial technology company, not a bank. Checking and banking services provided by Column N.A., Member FDIC.

See Brex in action

## Spend smarter. Move faster.

## Gain control _and_ speed with modern cards, banking, expenses, accounting, and more — in 120+ countries.

Get started

See Brex in action

...

Interface text:
Brex
Products
Solutions
Resources
Customers
Pricing
Sign in
See a demo
Get started
Enter your work email
AIE
aws

### Audio and video

11x chose Cloudglue to process audio and video because it can extract information from the video itself, beyond the audio transcript.

[12:20](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=740s) [Cloudglue supports audio and video and extracts information from video beyond its transcript.](https://assets.ai.engineer/sites/aiecode2025/sha256/04af3a86d5492f13da1304f0afd87a42cb44e7f0c3a03942e54f20aaf3079e2f.webp)

Audio & video

Cloudglue

Reasons:
• Supports both audio and video
• Capable of extracting info from video itself - not just transcript

Parsing
↓
Chunking
↓
Storage
↓
Retrieval
↓
Visualization

Interface text:
AIE

### Video parsing example

Cloudglue turns video files into Markdown with a summary, scene descriptions, visual content, and speech. The example shows a headphone review alongside its parsed document.

[12:31](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=751s) [A headphone review video is converted into a document with a summary, scenes, visual descriptions, and speech.](https://assets.ai.engineer/sites/aiecode2025/sha256/67cc03bd5323e5833cd761b5a670873a8387b912d07b99cc1ef4417ceba735d1.webp)

Parsing
Chunking
Storage
Retrieval
Visualization

# Video Document

## Title
Sony WH-1000XM6 & WH-1000XM5 Headphones Review and Comparison

## Summary
This video provides an in-depth review and comparison of Sony's flagship wireless noise-cancelling headphones, the WH-1000XM6 and WH-1000XM5. The presenter discusses design changes, comfort, sound quality, noise cancellation, microphone improvements, and portability, highlighting the XM6's stronger bass and smaller folding case. The video also covers the updated Sony Headphones app features, including ambient sound control and listening modes. Additionally, comparisons with other premium headphones like the Microsoft Surface Headphones 2+ and Apple AirPods Max are made, along with pricing considerations. The review concludes by affirming the WH-1000XM6 as the new leader in premium wireless headphones.

## Scenes

### Scene [00:00 - 00:20]

#### Sony WH-1000XM6 Headphones Review

A review of the Sony WH-1000XM6 wireless noise cancelling headphones, also known as the XM6 or the Mark 6.

Visual Content:
- [00:00 - 00:20]
- The video begins with a black and white abstract geometric animation.
- The scene transitions to a black and red abstract geometric animation on a white background.
- A man in a blue hoodie stands in front of a window, holding a pair of white headphones.
- He is centered in the frame and looks directly at the camera.
- The camera angle shifts to an overhead shot of a blue surface with various small objects scattered around.
- A person's hands enter the frame, opening a gray carrying case to reveal the white headphones inside.

Speech:
- [00:02 - 00:03] Okay.
- [00:03 - 00:08] So Sony is back with their new flagship wireless noise cancelling headphones with this super easy to remember name.

...

Interface text:
Sony WH-1000XM6 Review: Back on the Throne!
Marques Brownlee
20M subscribers
0:25 / 9:16
Join
Subscribe

AIE
Microsoft
smol ai

### Chunking

The speakers aim to break Markdown into semantic entities for embedding in a vector database while preserving meaningful structure, such as titles and paragraphs.

[12:38](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=758s) [Section slide titled Chunking.](https://assets.ai.engineer/sites/aiecode2025/sha256/b20e8eb4afc297d8a1ceea3fff21aa2d9bd2afb8b9f6a38f21f1db568f3c24d1.webp)

Chunking

Interface text:
AIE
Microsoft
smol ai

### Chunking

The team splits Markdown into semantic entities for embedding and storage in a vector database, while preserving structure such as titles and paragraphs.

[12:44](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=764s) [Section slide titled Chunking.](https://assets.ai.engineer/sites/aiecode2025/sha256/fff06bd08122f417860825c9ec36b0f5049fa048f9c63288b5e4ce983c1fbf4e.webp)

Chunking

AIE
aws

### What is chunking?

After parsing, chunking breaks Markdown into semantic units for embedding and storage, while preserving structure such as titles and paragraphs for later retrieval.

[12:47](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=767s) [Title slide asking what chunking is, with chunking highlighted in a processing pipeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/70ea7653f046a06306f65597ed4df0186375bff228bf8f6915571d14156e64ab.webp)

What is chunking?

Parsing
↓
Chunking
↓
Storage
↓
Retrieval
↓
Visualization

AIE

### What is chunking?

Chunking divides Markdown into logical pieces while preserving structure such as titles and paragraphs. The pieces can then be embedded, stored in a vector database, and retrieved later.

[12:51](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=771s) [Chunking is defined as splitting long blocks of text into logical chunks.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a5b3fe26fc47094ce204d0bfae5f391fd967d4713013711131661e92464a1f9.webp)

What is chunking?

Splitting long blocks of text into logical “chunks”

Parsing
Chunking
Storage
Retrieval
Visualization

AIE
Microsoft
smol ai

### What is chunking?

Chunking splits long documents into logical sections that can be embedded, stored in a vector database, and retrieved when relevant.

[13:17](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=797s) [Chunking splits text into logical chunks for vector database retrieval of relevant document sections.](https://assets.ai.engineer/sites/aiecode2025/sha256/ccfaa0a3a01e46d636cced42dafbec5418a11c7cab3940d714ac87c0b68be798.webp)

What is chunking?

Splitting long blocks of text into logical “chunks”
Recommended when using a vector DB
Allows you to retrieve relevant parts of a larger document

Parsing
Chunking
Storage
Retrieval
Visualization

AIE
Microsoft
smol ai

### Chunking strategies

Chunking sits between parsing and storage. Documents can be split by token or sentence counts, Markdown headers, an LLM, or a combination of these methods.

[13:34](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=814s) [Chunking strategies include splitting after a chosen number of tokens or sentences.](https://assets.ai.engineer/sites/aiecode2025/sha256/6791f8f4e5cf709bfcb5721c51cb71ba93dff3f9138541530d57b6a1dedab5db.webp)

Chunking strategies

• Split after n tokens
• Split after n sentences

Parsing → Chunking → Storage → Retrieval → Visualization

AIE

[13:35](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=815s) [Chunking strategies include splitting by tokens, sentences, Markdown headers, an LLM, or a combination.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce6a8df78c655a1ca75de9c83ec7366fa18e23cd2081f735274111365c127d3b.webp)

Chunking strategies

• Split after n tokens
• Split after n sentences
• Split on markdown headers
• LLM
• Any combination of the above

Parsing → Chunking → Storage → Retrieval → Visualization

AIE
Microsoft
smol ai

### Questions to ask yourself

Choose a chunking strategy around the logical units you need to preserve, the types of resources you have, and the queries you expect. Chunks must also be suitable for embedding and storage.

[14:00](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=840s) [Three questions ask about logical units in data, keeping them intact, and using different strategies for different resource types.](https://assets.ai.engineer/sites/aiecode2025/sha256/11b3b0d8203b796325ed3a76c99ba2e81633f9a74eb8219e8f03da0acbb42e25.webp)

Questions to ask yourself

What are the “logical units” in my data?
Which strategy will keep them intact?
Should I use a different strategy for each resource type?

Parsing → Chunking → Storage → Retrieval → Visualization

AIE
Microsoft
smol ai

[14:10](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=850s) [Four questions about logical units, chunking strategy, resource types, and expected queries.](https://assets.ai.engineer/sites/aiecode2025/sha256/973b7ffde97ec8e0199a59617889d37209b5d4ced6823c81db234fda3d1ba146.webp)

Questions to ask yourself

What are the “logical units” in my data?
Which strategy will keep them intact?
Should I use a different strategy for each resource type?
What kinds of queries am I expecting?

Parsing → Chunking → Storage → Retrieval → Visualization

AIE
aws

### Our strategy

The team splits Markdown first by headers, then by sentences and tokens. This preserves logical units while preventing chunks from becoming too long.

[14:50](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=890s) [Split on Markdown headers, then after a number of sentences and tokens; the slide says this preserves logical units and prevents overly long chunks.](https://assets.ai.engineer/sites/aiecode2025/sha256/a73acb9d9ac2ce83b7a9de7ae8fc226aaf3a6f4c912a195877b1bda664e4633f.webp)

Our strategy

Combination:
• Split on markdown headers
• Split after n sentences
• Split after n tokens

Works well for long and short documents
Preserves logical units
Prevents super-long chunks

Parsing → Chunking → Storage → Retrieval → Visualization

AIE
Microsoft
smol ai

### Storage technologies

RAG can retrieve context from several kinds of storage. The team chose a vector database for similarity search, while graph, document, relational, key-value and object stores are other options.

[15:35](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=935s) [RAG storage options include graph, document and relational databases, key-value stores, and object storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5c5a6bcb0796dc25184879f5780fd7cf2411bbc3c7d15d73231ddbc66e92ee0.webp)

Storage technologies

RAG != vector DB

Other options:
• Graph DBs (Neo4j)
• Document DBs (Elasticsearch)
• Relational DBs (MySQL)
• Key-value stores (Redis)
• Object storage (S3)

Pipeline:
Parsing → Chunking → Storage → Retrieval → Visualization

AIE
Microsoft
smol ai

### Vector DBs

The team used a vector database for similarity search. The slide shows a non-exhaustive selection of options, including Pinecone, Elastic, Milvus and PostgreSQL.

[15:49](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=949s) [Logos for Elastic, Milvus, Qdrant, OpenSearch, PostgreSQL, Pinecone, Weaviate, and Chroma.](https://assets.ai.engineer/sites/aiecode2025/sha256/6215feea75c4157019d33ee65b30d9915160f855adab5ec6a4905b21b5509633.webp)

Vector DBs

Elastic
Milvus
Qdrant
OpenSearch
PostgreSQL
Pinecone
Weaviate
Chroma

Parsing
Chunking
Storage
Retrieval
Visualization

AIE
Microsoft
smol ai

### Our choice

The team chose Pinecone for storage: cloud hosting avoided extra infrastructure, and bundled embedding models avoided a separate provider. Reputation, ease of getting started, and customer support also influenced the choice.

[16:00](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=960s) [Pinecone is the chosen vector database; the visible reason is that it is well-known.](https://assets.ai.engineer/sites/aiecode2025/sha256/06a214b57190a050895b9a89b5e094f6958815f923d86b77071db2e5c4a4bacf.webp)

Our choice

Pinecone

Reasons:
• Well-known

Parsing
Chunking
Storage
Retrieval
Visualization

Interface text:
AIE

[16:44](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1004s) [Pinecone was chosen for its reputation, cloud hosting, ease of getting started, embedding models, and customer support.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4cd10685ca581d8f1c35b801cb2d8c50285fc494c1c2d447683e564687efeef.webp)

Our choice

Pinecone

Reasons:
• Well-known
• Cloud-hosted
• Easy to get started
• Embedding models
• Good customer support

Parsing
Chunking
Storage
Retrieval
Visualization

Interface text:
AIE
aws

### RAG evolution

The slide uses Pokémon evolutions to compare traditional RAG, agentic RAG, and deep research. Deep research agents plan one or more retrieval steps and can decide whether to retrieve more context.

[17:46](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1066s) [Three illustrated stages: traditional RAG, agentic RAG, and deep research.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3b436410a095e45d397568cec010a7ee82095c030bdd18724ca004076708b17.webp)

RAG evolution

Traditional RAG
Agentic RAG
Deep Research

Parsing
Chunking
Storage
Retrieval
Visualization

### Our deep research agent

The Letta agent takes lead information, plans retrieval steps, and uses Pinecone to answer relevant questions. It returns the results as a list of questions and answers.

[18:17](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1097s) [Letta agent accepts lead information, formulates questions, generates answers using Pinecone, and outputs the questions and answers.](https://assets.ai.engineer/sites/aiecode2025/sha256/c53c26c009270dbfb4dbf8e4e6c1c7c4c563716b8c4f2c8410b064a6d1afbb3b.webp)

Our deep research agent

Letta

How it works:
1. Accepts lead info as input
2. Formulates list of relevant questions
3. Generates answers to each question using Pinecone tool
4. Outputs list of questions and answers

Parsing
Chunking
Storage
Retrieval
Visualization

AIE

### Lead research workflow

The agent generates two questions and queries Pinecone for each. If a query returns insufficient context, it is refined and retried; otherwise, the answers are generated and compiled.

[18:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1103s) [A workflow generates questions, queries Pinecone, refines queries when context is insufficient, generates answers, and compiles the output.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f66a27a75b46495d26ba7036ee33cdd88add3c3d574bc1446a7b2a5d90c6748.webp)

Lead Research

Question Generation

Question
Query Pinecone
Enough Context?
No: Refine Query
Yes: Generate Answer

Question
Query Pinecone
Enough Context?
Yes: Generate Answer
No: Refine Query

Compile Final Output

Parsing
Chunking
Storage
Retrieval
Visualization

AIE
Microsoft
smol ai

### Visualization

The speakers turn to visualization, framing it around customers’ need to trust Alice’s knowledge of their business.

[18:29](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1109s) [Visualization section title over a landscape.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ecc195184b0bef35b58874824c92c6a1c3c0e403a45c8a0ce8515f3722185be.webp)

Visualization

AIE
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=0s) [upbeat music] Okay.

[0:15](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=15s) Thanks everyone for coming today. Uh, so today's talk is called Building Alice's Brain: How We Built an AI Sales Rep That Learns Like a Human.

[0:25](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=25s) Uh, my name is Sherwood. I am one of the tech leads here at 11x. I lead engineering for our Alice product, and I'm joined by my colleague, Satwik.

[0:34](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=34s) So 11x, for those of you who are unfamiliar, is a company that's building digital workers for the go-to-market organization. We have two digital workers today. We have Alice, who is our AI SDR, and then we also have, uh, Julian, who is our voice agent, and we have more workers on the way.

[0:49](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=49s) Today, we're gonna be talking about Alice specifically, and actually, uh, Alice's brain or the knowledge base, which is effectively her brain.

[0:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=58s) So let's start from the basics. Uh, what i-- what is an SDR? Well, an SDR is, is a sales development representative, if you're not familiar. I know that's a room full of engineers, so I s-- thought I would start with the basics.

[1:08](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=68s) And this is essentially an entry-level sales role. This is the kind of job that you might get, uh, right out of school, and your responsibilities, uh, basically boil down to three things.

[1:17](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=77s) First, you're sourcing leads. These are people that you'd like to sell to. Then you're contacting them or engaging them across channels. And finally, you're booking meetings with those people.

[1:26](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=86s) So your goal here is to generate positive replies and meetings booked. These are the two, uh, key metrics for an SDR.

[1:34](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=94s) And a lot of an SDR's job boils down to writing emails like the one that you see in front of you right now. This is actually an email that Alice has written, and, uh, it's an example of the type of, uh, of, the type of work output that Alice has.

[1:47](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=107s) Uh, Alice sends about fifty thousand of these emails to, uh, uh, in a given day, and that's in comparison to a human SDR who would send twenty to fifty.

[1:55](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=115s) Uh, and Alice is now running campaigns for about three hundred different, uh, business organizations.

[2:03](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=123s) So before we go any further, I want to define some terms because, uh, since we work at 11x, we have our customers, but then our customers also have their customers, so things get a little confusing.

[2:13](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=133s) Uh, today we'll be using the term seller to refer to the company that is selling something through Alice, that is our customer, and then we'll be using the term lead to refer to the person who's being sold to.

[2:24](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=144s) And here's what that looks like as a diagram. You can see the seller is pushing context about their business. These are the, the products that they sell or the, uh, case studies that they have that they can reference in emails.

[2:34](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=154s) She-- they're pushing that to Alice, and then Alice is then using that to personalize emails for each of the leads that she contacts.

[2:42](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=162s) So there are two requirements that Alice needs to, uh, in order to succeed in her role. The first is that she needs to know the seller, the products, the, the services, the case studies, the pain points, the value props, the ICP.

[2:54](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=174s) And the second is that she needs to know the lead, uh, their role, their responsibilities, what they care about, what other solutions they've tried, uh, pain points that they might be experiencing, the company they work for.

[3:05](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=185s) And today, we're gonna be really focused on knowing the seller.

[3:09](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=189s) So in our-- in the old version of our product, the seller would be responsible for pushing context about her, uh, about their business to Alice, and they did so through a manual experience, uh, called the library.

[3:22](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=202s) And here you can see what it looks like there, where the library shows, uh, all of the different products and offers that are available for this business that, uh, Alice can then reference when she writes emails.

[3:32](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=212s) The user would have to enter details about all, every individual product and service and all of the pain points and solutions and value props associated with them in our dashboard and including these detailed descriptions.

[3:43](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=223s) And those descriptions would, uh, were, were important to get right because these actually get included in the context for the emails or for Alice when she writes the emails.

[3:52](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=232s) Then later on, during campaign creation, uh, this is what it looks like to, to create a campaign. And you can see we have a lead in the top left, and the user is selecting the different offers that they've defined from the library in the top right that...

[4:04](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=244s) And these are the offers that Alice has access to when she's generating her emails.

[4:09](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=249s) We had a lot of problems with this user experience, and the first one was it was just extremely tedious. It was a really bad and, and cumbersome user experience.

[4:17](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=257s) The user had to enter a lot of information, and that created this onboarding friction where, uh, users couldn't actually run campaigns until they had filled out their library.

[4:27](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=267s) And finally, the emails that we were generating using this approach were just suboptimal. Users would have to either choose between too few ema-- uh, too few offers, uh, which meant that, uh, you'd have irre-irrelevant offers for a given lead or too many offers, which means that you have all of the stuff in the context window, and Alice

[4:43](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=283s) just wasn't as smart when she write the, writes those emails.

[4:47](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=287s) So how can we address this? Well, we had an idea, which is that instead of the seller being responsible for pushing context about the business to Alice, we could flip things around so that Alice can proactively, uh, pull all of the context about the seller into her system and then use whatever, whatever is most relevant when writing

[5:05](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=305s) those emails. And that's effectively what we accomplished with the knowledge base, which we'll tell you more about in just a moment.

[5:12](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=312s) So for the rest of the talk, we're going to first do a high-level overview of the knowledge base and how it works. Then we will do a deep dive on the pipeline, the different steps in our, uh, RAG system pipeline.

[5:24](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=324s) Then after that, we will talk through the user experience of the knowledge base, and we will wrap up with some lessons from this project and, uh, future plans.

[5:33](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=333s) So let's start out with, uh, an overview.

[5:35](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=335s) All right. So overview, what is knowledge base, right? It's basically a way for us to kind of get closer to a human experience. Like if a hum-- if you're training a human SDR, you would kind of get them in, and then you will basically dump a bunch of documents on them, and then they ramp up throughout a

[5:53](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=353s) period of like weeks or months, um, and you can basically check in on their prog-progress. Um, and similar to that, knowledge base is basically a centralized repository on our platform for the seller info, and then users can kind of come in, dump all their source material, and then we are able to reference that information at the time

[6:11](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=371s) of message generation.

[6:13](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=373s) Um, now what resources do SDRs care about? Here's a little glimpse into that. Marketing materials, case studies, uh, sales calls, press releases, you know, and a bunch of other stuff.

[6:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=383s) Um, now how do we bucket these into categories that we're actually going to parse? Uh, well, we created documents and images, websites, and then media, audio, video, and you're gonna see why that's important.

[6:36](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=396s) So here's an overview of what the architecture looks like. It starts off with the user uploading something, any document or resource in the client, and then we save it to our S3 bucket and then send it to the back end, um, which then, you know, creates a bunch of resources in our DB and then kicks off a

[6:53](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=413s) bunch of jobs depending on the resource type and the vendor selected. Now, the vendors are asynchronously doing the parsing. Once they're done, they send a webhook to us, which we consume via ingest.

[7:03](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=423s) And then once we've consumed that webhook, we take that parsed, uh,

[7:09](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=429s) uh, artifact that we get back from the ven-vendors, and then we store it in our DB and then at the same time upsert it to Pinecone and embed it.

[7:17](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=437s) Um, and then eventually once we store it in the local DB, we have like a UI update, and then eventually our agent can query Pinecone, our vector DB, for that stored information that we just put in.

[7:31](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=451s) So now that we have a high level of understanding of how the knowledge base works, let's dig into each individual step in the pipeline. There are five different steps in the pipeline.

[7:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=459s) The first is parsing, then there's chunking, then there's storage, then there's retrieval, and finally, we have visualization, which will, uh, sounds a little untraditional, but we'll cover it in a, in a moment.

[7:52](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=472s) So let's start with parsing. Uh, what is parsing? I think that we probably all take this for granted, but it's worth defining. Parsing is the process of converting a non-text resource into text.

[8:02](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=482s) And the reason that this is necessary is because, as we all know, language models, they speak text. So in order to make information that is represented in a different form like a PDF or an MP4 file or a, or an image legible or useful to the LLM, we need to first convert it to text.

[8:19](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=499s) And so one way of thinking about parsing is it's the process of making non-text information legible to a large language model. Um, and we do have multimodal models that are one solution to this, but there are lots of restrictions on multimodal models that make it, um, that make parsing still relevant.

[8:35](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=515s) So to illustrate that, we have the five different document types or resource types that we mentioned momentarily ago, uh, going through our parsing process and coming out as actually Markdown, which is a type of text that, uh, as we all know, contains, uh, some structural information and, and formatting, which is actually semantically, semantically meaningful and useful.

[8:53](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=533s) Let's talk about the process of how we implemented parsing, and the, the short answer is that we did not. We didn't wanna build this from scratch, and we had a few different reasons for doing this.

[9:02](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=542s) The first is that w- you just saw that we had five different resource types and a lot of different file types within each of them. We thought it was gonna be too many, and we thought it was gonna be too much work.

[9:11](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=551s) We wanted to get to market quickly. Um, the last reason was that we just weren't that confident in the outcome. There are vendors who dedicate their entire company to building an effective parsing system for a specific resource type.

[9:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=563s) We didn't want our team to, to have to become specialists in, in parsing for each one of these resource types and to build a, a parsing system for that.

[9:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=570s) We thought that maybe if we tried to do this, the outcome actually just wouldn't be that, that successful. So we chose to work with a vendor, and here are a bunch of the vendors that we, uh, we came across.

[9:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=581s) You can find ten or twenty or fifty with just a quick Google search, but these are some of the leaders that we evaluated.

[9:48](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=588s) And in order to make a decision, we came up with some requir-requirements, and it was three specific requirements. The first was that we needed, uh, support for our necessary resour-resource types.

[9:57](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=597s) That goes without saying. We also wanted Markdown output. And then finally, we wanted this vendor to support webhooks. We wanted to be able to receive that output in a convenient manner.

[10:08](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=608s) A few things that we didn't consider to start out with. Accuracy. Crazy, we didn't consider accuracy. We didn't consider either accuracy or comprehensiveness. Our assumption here was that most of the vendors that are leaders in the market are gonna be within a reasonable band of accuracy and comprehensiveness.

[10:25](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=625s) And accuracy would refer to whether or not the extracted output is-- actually matches the, the original resource. Comprehensiveness, on the other hand, is the amount of extracted information that is, uh, available, um, in the, in the final output.

[10:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=639s) And the last thing that we didn't really consider was cost, uh, to be honest, and this was because we were-- the system was pre-production. We didn't have real production data yet, and we didn't know, uh, what our usage would be.

[10:50](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=650s) And so we, we figured what we would do is we'd come back and optimize cost once we had real usage data.

[10:56](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=656s) So onto our final selections. For documents and in-images, we chose to work with LlamaParse, which is a LlamaIndex product. Uh, I think Jerry was up here earlier today. Uh, and the reasons that we chose to work with LlamaParse was, first, it supported the most number of file types out of any document parsing solution we could find, and

[11:15](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=675s) second, their support was really great. Jerry and his team were, were, were quick to get in a Slack channel with us, I think within just a couple of hours of us doing an initial evaluation.

[11:25](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=685s) And with LlamaParse, we're able to turn documents like this PDF of a s-- uh, 11x sales deck into a Markdown file like the one you see on the right.

[11:34](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=694s) For websites, we chose to work with Firecrawl. The other main vendor that we were considering was Tavily, and this is actually not really a m- a major knock on Tavily.

[11:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=701s) For Firecrawl, we chose to work with them because first we were familiar. We had already worked with them on a previous project. And secondly, Tavily's crawl endpoint, which is the endpoint that we would have needed for this project, was still in development at the time, so it wasn't something we could actually use.

[11:57](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=717s) And similar to, uh, LlamaParse with Ta-- with, uh, Firecrawl, we are able to take a website like this Brex homepage that you see here and turn it into another Markdown document.

[12:07](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=727s) Then we have audio and video. And for audio and video, we chose to work with a newer, uh, upstart vendor called Cloudglue.

[12:14](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=734s) And the reasons that we chose to work with Cloudglue were first, they supported both audio and video, not just audio. And con-- second, they were actually capable of extracting information from the video itself, as opposed to just transcribing the video and giving s- us back, uh, a, a Markdown file that contains the transcript of the audio.

[12:32](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=752s) And so with Cloudglue, we're able to turn, uh, YouTube videos and MP4 files and other video formats into Markdown like you see on the right.

[12:39](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=759s) So now that everything is Markdown, we move on to the next step, which is chunking.

[12:43](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=763s) All right. Markdown, let's go. Now, basically, we have a blob of Markdown, right? And we want to kind of break it down into, like, semantic entities that we can embed and put it in our vector DB.

[12:56](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=776s) At the same time, we wanna, uh, protect the structure of the Markdown because it contains some meaning inherently, like something's a title versus something's a paragraph. There is inherent meaning behind that.

[13:08](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=788s) Um, so we're splitting these long blobs of text, like 10-page documents, into chunks that we can eventually retrieve, uh, after we've embedded and stored them in a vector DB, right?

[13:18](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=798s) And now basically, we can im- like, take all of this and, like, we're thinking about how we can s- you know, split a long document into chunks, right? So chunking strategies.

[13:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=810s) Um, you have various things that you can do. You can split on tokens, you can split on sentences. You can also split on Markdown headers, right? And then you can do like LLM calls and have an LLM split your [chuckles] document into chunks, you know, or any combination of the above.

[13:46](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=826s) Um, now, what you wanna ask yourself when you're deciding on a chunking strategy is like, um, what kind of logical units am I trying to preserve in my data, right?

[13:55](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=835s) What do I eventually want to extract during my retrieval, right? What strategy will keep them intact, and at the same time, you're able to successfully embed them and store them in whatever DB you want.

[14:06](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=846s) Um, so and then should I s- try a different strategy for different resource types? We have like, we have to deal with PDFs, PowerPoints, videos, right? Um, and then eventually, what kinds of queries or retrieval strategies am I expecting?

[14:20](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=860s) Um, and then we ended up with like a combination of all the three, like, all the things that we mentioned. So we split on Markdown headers, and then we kind of a waterfall.

[14:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=870s) So because we want our, like, records in our vector DB to be a certain token count, so we split on Markdown headers, and then we split on sentences, and then eventually we split on tokens.

[14:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=881s) And then, yeah, it's like worked well for us, uh, for all types of documents. Um, and it has successfully preserved our Markdown chunks that we can kind of cleanly show in the UI.

[14:51](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=891s) Um, and it also prevents super long chunks, which are, you know, diluting the meaning behind your document if you end up with that.

[14:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=898s) Okay, so we have split all of our Markdown into individual chunks. It's now time to put those chunks somewhere. We're gonna store them. Let's talk about storage technologies. So for storage technologies, I'm sure everyone is, like, here for the RAG section, so they think that we're using a vector database.

[15:12](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=912s) We actually are using a vector database. But to be pedantic, RAG is Retrieval-augmented generation, so we all know that, uh, anytime you're retrieving context from an external source, whether it's a graph database or Elasticsearch or, uh, a file in the file system, that also qualifies as RAG.

[15:29](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=929s) Um, some of the other options you can use for RAG, uh, I just mentioned a graph database, document databases, uh, relational databases, key-value stores. You could even use object storage like S3.

[15:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=941s) In our case, we did use a vector database, and that's because we wanted to do simi- similarity search, which is what vector databases are, are built for and optimized for.

[15:51](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=951s) Once again, we had a lot of options to choose from. This is, uh, not a complete or an exhaustive list.

[15:58](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=958s) In the end, we chose to work with a company called Pinecone, and the reason that we chose to work with Pinecone was first, it was a well-known solution. We were kind of new to the space, and we thought, probably can't go wrong going with the market leader.

[16:11](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=971s) It was cloud-hosted, and so our team wouldn't have to spin up any additional infrastructure.

[16:16](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=976s) It was really easy to get started. They had great getting started guides and SDKs.

[16:21](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=981s) Uh, they had embedding models bundled with the solution. So for a vector database, typically you have to embed the information before it goes into the database. Uh, that would require the use of a third-party or an external vector da- vector, uh, excuse me, embedding model.

[16:33](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=993s) And, uh, with Pinecone, we didn't actually have to go find another embedding model provider or host our own embedding model. We just used the one that they provide. And last but not least, their customer support was awesome.

[16:43](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1003s) They got on a lot of calls with us, helped us analyze, uh, different vector datas- database options and think through, uh, graph databases and graph RAG, whether that made sense for us.

[16:54](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1014s) So retrieval, the RAG part of the RAG workflow that we just built, right? Um, you'll see that there's actually an evolution of different RAG techniques over the last year.

[17:04](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1024s) We started off with just traditional RAG, which is kind of a play on you're pulling information and then enriching your system prompt for an LLM API call, right? And then eventually, that turned into an agentic RAG form, where now you have all these tools for getting information retrieval, and then you attach those tools to whatever agentic flow

[17:24](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1044s) that you have, and then it calls the tool as a part of its larger flow, right? Now, something we- we're seeing emerge in the last couple of months is deep research RAG, where now you have these deep research agents which are coming up with a plan, and then they execute them, and the plan may contain one or

[17:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1061s) many steps of retrieval, right? These deep research agents can go broad or deep depending on the context needs, and they can evaluate whether or not they wanna do more retrieval.

[17:51](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1071s) Um, we ended up building a deep research agent. Um, we actually used a company called Letta. Letta is a cloud agent provider, and they're really easy to build with.

[18:02](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1082s) Um, how it works, basically, we pass in the lead information to our agent, and then it basically comes up with a plan. Plan contains one or many context retrieval steps, and then eventually You know, does the tool call, summarizes the results, and then generates an answer for us in a nice clean Q&A manner, right?

[18:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1103s) And then this is kind of how it looks like for a system with two questions that we ask. Um,

[18:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1110s) Vis?

[18:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1110s) Now on to visualization, the most, uh, mysterious part of the pipeline. So what does visualization have to do with a, a RAG or ETL pipeline? Um, for more context, our customers are trusting Alice to represent their business.

[18:44](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1124s) They really want to know that Alice knows her stuff, that she actually knows the products that they sell, and she's not gonna lie about case studies or testimonials or, uh, make things up about the pain points that they address.

[18:53](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1133s) So how can we reassure them? In our case, we came up with a solution which is to let Al- let users peek into Alice's brain. Get ready.

[19:02](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1142s) This is what that looks like. We have a, an interactive three-D visualization of the knowledge base available in the product. What we've done here is taken all of the vectors, vectors from our, uh, Pinecone vector database and, uh, collapsed, or actually, excuse me, I think the correct term is projected them down to just three dimensions.

[19:21](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1161s) So we're gonna render them as nodes in three-dimensional space.

[19:23](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1163s) UMAP.

[19:24](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1164s) Um, [chuckles] UMAP. With using UMAP. Uh, and once the nodes are visible in this space, you can click on any given node to view the associated chunk. This is one of the ways that, uh, for example, our sales team, our support team will demonstrate Alice's knowledge.

[19:40](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1180s) Now how does it look like in the actual UI, right? Basically, you start off with this nice little modal. You know, you drop in your URLs, your web pages, your documents, your videos, and then you click Learn, and then it kinda shows up nicely in the UI.

[19:53](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1193s) Um, you have all the resources there, and then you have the ability to interrogate Alice about what she knows of your knowledge base, right? It's a really nice agent that we built, again, using Letta.

[20:04](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1204s) And here's how it looks like in the campaign creation flow. You see that on the left-hand side we have the knowledge base content showing up as a nice Q&A where you can click on the questions and it shows you a dropdown of the chunks that we retrieve, and these were used as a part of the messaging flow.

[20:21](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1221s) So now with that, we have achieved our goal. Our agent is closer to a human than being an email-a-tron, right? We are now, [laughs]

[20:30](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1230s) we are now basically, uh, emulating how you onboard a human SDR. You dump in a bunch of context, and they just know.

[20:38](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1238s) So in conclusion, the knowledge base was a pretty revolutionary project for our product and really changed the user experience and also leveled up our team a lot. Uh, we learned a lot of lessons.

[20:48](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1248s) It was hard to create this slide, but there are just three that I wanna highlight for you today. The first was that RAG is complex. It was a lot harder than we thought it was gonna be.

[20:57](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1257s) There were a lot of micro-decisions made along the way, a lot of different technologies we had to evaluate. Supporting different resource types was hard. Hopefully you all have a better appreciation of how complicated RAG can be.

[21:08](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1268s) The second lesson was that you should first get to production before benchmarking, and then you can improve. And the idea here is that with all of those decisions and vendors to evaluate, uh, it can be hard to get started.

[21:19](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1279s) So we recommend just getting something in production that satisfies the product requirements and then establishing some real benchmarks which you can use to iterate and improve.

[21:28](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1288s) And the last learning here was that you should lean on vendors. You guys are all gonna be buying solutions, and they're gonna be fighting for your business. Make them work for it.

[21:36](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1296s) Make them teach you about the different, uh, the different offerings and why their solution is better.

[21:41](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1301s) And so our future plans are to first track and address hallucinations in our emails,

[21:47](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1307s) evaluate parsing vendors on accuracy and completeness, those, uh, metrics that we, uh, identified earlier, experiment with hybrid RAG, the introduction of a graph database alongside our vector database, and finally to just focus on reducing cost across our entire pipeline.

[22:03](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1323s) And if any of this sounds interesting to you, we are hiring, so please reach out to either Satwik or myself.

[22:07](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1327s) Join us.

[22:09](https://www.youtube.com/watch?v=KWmkMV0FNwQ&t=1329s) And, uh, thank you all for coming today. [audience applauding] [upbeat music]
