# Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

Jacob Lauritzen, Legora · Simon Eskildsen, turbopuffer

[Talk page](https://ai.engineer/talks/V-isu4eTHgw-connect-ai-billions-legal-documents) · [Video](https://www.youtube.com/watch?v=V-isu4eTHgw)

## Connect AI to Billions of Legal Documents

Legora’s migration from Elasticsearch through Postgres to turbopuffer shows how project lifecycles, customer encryption requirements and storage round trips shape a search engine for billions of legal documents.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 416 seconds](https://ai.engineer/api/talk-essay-media/102bc76027ec84b3a6dc161743e25baa4f48cfc7975e99cd55fd1dce8394aec3)

Making the project the unit of legal search

### Two kinds of legal search

[0:13](https://www.youtube.com/watch?v=V-isu4eTHgw&t=13s)

A lawyer reviewing an acquisition needs to search the documents for that particular deal. A lawyer researching a dispute may need to search laws, cases and regulations, then follow relationships between them. Legora serves both workloads, alongside contract review, drafting and collaboration. Jacob Lauritzen, a Legora engineer, introduces the application; [Simon Eskildsen](https://sirupsen.com/about), turbopuffer’s CEO and co-founder, explains the search engine underneath it.

- **Project search:** Lauritzen imagines SpaceX acquiring Cursor. The law firm handling this hypothetical deal uploads employment agreements and supplier contracts into one project, then searches within that collection. Projects range from tens of documents to millions, but the scope stays clear: this matter’s documents.

- **Legal research:** The collection spans laws, previous cases and regulations. Lawyers use it to answer legal questions or support litigation. Finding relevant material also means understanding which sources apply and how they relate.

The project is already the application’s unit of work. The first part of the migration is about making it a useful unit of storage, too. That matters because a project can be intensely active for a while, then close and almost never receive another query.

### Residency and enterprise isolation multiply the infrastructure

[2:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=156s)

Legora’s project-search journey runs from hundreds of thousands of documents to two billion. Initially, shared blob storage held the raw documents and one **Elasticsearch** cluster indexed and searched them for every tenant. Lauritzen says the arrangement worked reasonably well: it was simple, and the team had one search deployment to operate.

Customers in different regions wanted their processing to stay local. Legora reproduced its setup across the EU, US and Asia Pacific to meet requirements from European, American and Australian customers. The design still worked, but each regional copy brought more infrastructure to run. Residency changed the deployment topology before document volume forced a different search architecture.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 225 seconds](https://ai.engineer/api/talk-essay-media/786be716d23afe7b66e3a5f7c7c0cccff33283529bc3dd6dc5cf5a9317ae83b1)

A simple cluster becomes several systems

Large banks and law firms added two further requirements:

- **Physical isolation:** Customers wanted their data separated, which Lauritzen describes here as effectively asking for their own database. He acknowledges that customers can mean different things by physical isolation.

- **Customer-managed encryption keys:** A customer keeps an encryption key in its key vault and grants Legora access. Legora uses the key to encrypt and decrypt data at rest. Revoking access prevents Legora from using that key for further decryption, giving the customer control over access to the encrypted stored data.

These demands made a shared search cluster harder to fit into the rest of the system. Legora already had to separate its transactional databases and blob storage, so the next move tried to consolidate search into those existing databases.

### Why 4,000 partitions still thrashed the cache

[4:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=288s)

Moving search into **Postgres** had a practical appeal. Legora already ran it for transactional workloads and already needed multiple database and blob-storage arrangements. Adding vector and text search meant fewer kinds of systems to operate. The setup used `pgvector` for vector search and `tsvector` for text search. Text ranking no longer used **BM25**, and Lauritzen reports a loss in retrieval quality.

### Hashing projects separated keys, but mixed their lifecycles

The team split its document-chunk table into roughly 4,000 partitions, hashed each project key and packed projects into those partitions. This spread projects across the database without keeping each project independently stored. Active projects and abandoned projects could land together. As the corpus grew, the partitions accumulated large amounts of data that almost never needed searching.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 332 seconds](https://ai.engineer/api/talk-essay-media/fbd3b58b31533f3a3afaaa010b638ae5d33bf90ab86e49b56cb7b0346c396b4d)

Why 4,000 partitions still thrashed the cache

Lauritzen describes a query bringing partition data into memory, followed by another query loading a different partition and displacing the previous data. The cache kept making room for large collections containing both useful active data and inactive data. Repeated queries therefore failed to benefit from a stable working set. Search and ingestion P99 latency rose from about 100 milliseconds to 20 seconds.

The failure was specific to the packing arrangement and access pattern. Increasing the partition count had distributed the data, but a lawyer’s project remained only part of a larger storage collection. A cold project could still compete for cache space whenever an active neighbor was searched.

### One namespace per project

Legora moved to **turbopuffer** at approximately 400 million documents, a size Lauritzen gives as an estimate. Each project received its own namespace. An unused project could remain in blob storage while an active project was cached and searched independently. The storage unit now followed the application’s unit of work.

The migration also restored BM25, improved relevance and latency, lowered cost and let Legora operate a single turbopuffer cluster instead of its collection of separate search databases. These benefits arrived together with the new storage layout; they do not isolate the effect of any one indexing algorithm. The clear structural improvement was that inactive projects no longer had to share large partitions with active ones.

What a project query brings into the working set
Hash-packed partitions coupled active projects to inactive neighbors. Project namespaces let inactive collections remain in object storage.
- Active and inactive projects — Project keys hash into shared, growing partitions.
- Partition data in memory — Queries against other partitions displace the cached data.
- Active project namespace — The project is an independent search collection.
- Cached project data — Queries check caches for the requested namespace.
- Inactive project namespace — Unused projects remain in blob storage.
- Active and inactive projects → Partition data in memory (Load shared partition data)
- Active project namespace → Cached project data (Search this namespace)

### Direct object-storage writes make round trips the main read constraint

[7:28](https://www.youtube.com/watch?v=V-isu4eTHgw&t=448s)

Eskildsen begins with turbopuffer’s fundamental tradeoff: writes go directly to object storage, using S3 in his example. Turbopuffer does not first run its own disk-replication or Paxos path. Writes therefore take hundreds of milliseconds. He uses inventory reservations for a Kylie Jenner flash sale on Shopify as an example of a workload that would be a poor fit. Search ingestion can often accept that write delay if reads remain responsive.

A write enters a write-ahead log in object storage. Background workers build vector, text and columnar indexes to answer subsequent queries. Eskildsen pictures the log as successive files—`one.json`, `two.json`, `three.json`—while explicitly saying that JSON is an illustration. The useful model is a stored sequence of writes from which searchable structures are built.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 491 seconds](https://ai.engineer/api/talk-essay-media/fc3b0db57bea41a605b719f62828d4b8e317033c02f19253e64063e57a6cde9b)

Write to object storage, then make reads economical

### Route toward cached data, then descend the hierarchy

A namespace acts much like a table; Eskildsen compares it to an isolated directory on S3. Any read-replica node can serve its queries, but routing favors the node most likely to have its data cached. The node checks memory, then an NVMe SSD cache, then object storage. This affinity encourages reuse without making one node the sole owner capable of answering.

For cold data, the costly part is waiting through dependent storage requests. Eskildsen cites an S3 P99 of around 200 milliseconds for a one-megabyte blob and describes roughly three round trips as an ideal target. If each fetch must finish before the next can begin, those waits accumulate. Turbopuffer instead tries to do as much work as possible concurrently within a few rounds. The same design helps use modern disks efficiently.

That requirement reaches into the indexes and query planner: they must make useful progress with the data fetched in each round. Eskildsen’s [architecture article](https://turbopuffer.com/blog/turbopuffer) develops the same tradeoff between downloading more data per round and making additional requests. Object storage supplies cheap capacity; the search engine has to organize its work around the cost of reaching it.

Separate the write path from the cache-aware read path
Writes enter object storage directly. Reads favor a node with cached namespace data and fall through storage tiers on misses.
- Incoming write — Accept hundreds of milliseconds of write latency.
- Object-storage write-ahead log — Store the sequence of writes.
- Background index construction — Build vector, text and columnar indexes.
- Namespace query — Favor the replica likely to hold cached data.
- Memory cache — First place to look for requested objects.
- NVMe SSD cache — Next tier when enabled for the workload.
- Object storage — Fetch cold objects while limiting dependent round trips.
- Incoming write → Object-storage write-ahead log (Write directly)
- Object-storage write-ahead log → Background index construction (Build in the background)
- Namespace query → Memory cache (Check cached objects)
- Memory cache → NVMe SSD cache (On a memory miss)
- NVMe SSD cache → Object storage (On an SSD miss)
- Memory cache → Object storage (When SSD caching is disabled)

### Per-namespace encryption includes the cache question

[9:42](https://www.youtube.com/watch?v=V-isu4eTHgw&t=582s)

The namespace also gives turbopuffer a natural place to apply customer-specific storage and encryption settings:

- **Separate keys:** Each namespace can use a different encryption key, and data can be re-encrypted with another key.

- **Separate buckets:** Namespaces can live in different buckets, including buckets in customers’ own cloud accounts. Eskildsen describes customers using thousands of buckets.

- **Shared storage where permitted:** Namespaces can also share a bucket. The engine can change storage arrangements at namespace granularity rather than requiring a whole search deployment for each collection.

Legora needed its namespaces separately encrypted at rest. S3, GCS and Azure Blob Storage could meet the storage requirement, but the NVMe SSD cache created a disagreement. Turbopuffer treated that cache as volatile, like memory. Legora’s customers treated the SSD contents as data stored at rest. The word cache did not exempt those bytes from their encryption requirements.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 660 seconds](https://ai.engineer/api/talk-essay-media/f44307a74a733fc3b95d37acc25025429ccae5c7f6e82faa0ecde23272e5da14)

The namespace also becomes the unit of encryption

### Disable first, then measure

The team expected to implement encryption in the disk cache. Before doing that work, it disabled the cache and measured performance. Memory caching plus object storage performed well enough that some Legora workloads kept the SSD cache disabled. This was a useful simplification for those workloads; it does not establish that SSD caching is unnecessary in general. Encrypted disk-cache support remained future work in Eskildsen’s account.

Returning to project-search results, Lauritzen reports approximately an order-of-magnitude improvement in median latency and a larger improvement at P99. He gives no resulting absolute latency values in this explanation. The gains matter across an agent’s work: an agent making 20 or 100 searches repeatedly encounters retrieval delay, so search performance affects how much investigation it can complete within a useful response time.

### Legal research turns one question into a growing set of searches

[12:18](https://www.youtube.com/watch?v=V-isu4eTHgw&t=738s)

The newer legal-research corpus was growing toward ten billion vectors. Lauritzen presents that as a direction of growth, rather than an already reached total. The workload also creates spikes in queries per second: one user question fans out into several searches, and the investigation keeps generating more. A large corpus and a high query rate place different demands on the engine.

The searches expand because retrieving a similar passage does not settle whether it applies. Lauritzen identifies three relationships that require filtering and further retrieval:

- **Jurisdictional hierarchy:** Sources can sit at city, county, state or federal levels. Retrieval must respect the relevant hierarchy of authority.

- **Temporal validity:** A decision may have been overruled. Finding the decision creates a need to determine whether it remains valid.

- **Exceptions and special cases:** A newer regulation may create an exemption to an older one. Finding either source can require finding the other to understand the rule that applies.

These relationships make the research problem graph-like: a useful result points toward other material that must be checked. Lauritzen explains why the search expands, but does not detail an algorithm for determining legal authority or resolving every temporal relationship. The storage engine supports the retrieval work; the application still has to decide which searches and filters the legal question requires.

### Jurisdictions become namespaces

Legal research also started on Elasticsearch, and Legora was moving it to turbopuffer because keeping the whole corpus in Elasticsearch had become expensive. Here the namespace follows a jurisdiction. Lauritzen uses frequently queried EU law and less frequently queried Danish law to illustrate the hot and cold ends of the workload. The distinction describes access frequency in his example.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 829 seconds](https://ai.engineer/api/talk-essay-media/9eab8f8de9de4accc6864ffe1ab3a294a67ab57c763559c1d3e260b966b0cd2a)

Legal research makes one question become many searches

An infrequently queried jurisdiction can remain in blob storage until needed. Lauritzen says a 500-millisecond cold fetch is acceptable for this deep-research workload. That tolerance makes the long tail economical: keep busy collections closer to compute, and pay an occasional fetch delay for the many collections that rarely receive attention. The cost advantage depends on being able to accept those cold reads.

### Clustered vector trees put routing information in fast memory

[14:20](https://www.youtube.com/watch?v=V-isu4eTHgw&t=860s)

Eskildsen describes turbopuffer as puffing data into and out of the memory hierarchy. Frequently used data can justify DRAM, SSDs offer another balance of capacity and latency, and cold data stays in object storage. The goal is to keep as much data as possible in cheaper tiers while still meeting performance needs. Each tier changes the economics of random access, sequential access and waiting for a round trip.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 908 seconds](https://ai.engineer/api/talk-essay-media/107c8506eaa8fd9075cd8fa2966cf94ce0dcb0b247808036278a7b29e6f5a7c9)

Why clustered vectors suit the storage hierarchy

### Graph navigation creates dependent waits

For vector search, he contrasts navigating a graph with organizing vectors into clusters. In the graph example, the engine visits a node to discover where to go next. If each step requires an uncached object-storage request, the query can repeatedly encounter the roughly 200-millisecond S3 P99 he cited earlier. Shrinking the graph’s diameter helps reduce steps, but the access pattern still creates dependencies between fetches. A layout effective in memory can become expensive farther down the storage hierarchy.

Turbopuffer instead groups nearby vectors into clusters, groups those clusters into larger clusters, and repeats this to form a tree. Eskildsen likens it to a complicated B-tree over the geometry of the vector space. The comparison describes the hierarchy: approximate geometric clustering lets a query narrow down which parts of the collection to inspect.

The tree separates two kinds of data with different access frequencies:

- **Upper-level centroids:** Searches repeatedly consult these cluster representatives to choose a path through the index. Their frequent use makes them good candidates for DRAM.

- **Leaf data:** The much larger collection of underlying records need not all occupy DRAM. Eskildsen describes leaves on SSDs with an illustrative one-millisecond final round trip. Colder objects can remain in object storage.

The index’s shape therefore helps determine what stays cached. Compact routing information is useful across many searches, while any particular leaf is needed less often. Keeping them in different tiers saves fast-memory capacity without forcing every query to discover its route through a succession of cold reads.

Eskildsen calls this the cheapest way to run a database and extends the argument to customers indexing large portions of the web. That is his architectural judgment, rather than a universal result established by a comparative benchmark here. The mechanism is more specific and useful: even within one searchable collection, routing structures and underlying records have different access patterns and can justify different storage tiers.

Follow a vector query through cached cluster representatives
The tree repeatedly narrows the vector space using upper-level centroids, then reaches the selected leaf data.
- Upper-level centroids — Frequently consulted cluster representatives, mostly in DRAM.
- Clusters within clusters — Narrow the approximate geometric search through the tree.
- Selected leaf data — Larger underlying records can remain on SSDs rather than all occupying DRAM.
- Upper-level centroids → Clusters within clusters (Choose relevant clusters)
- Clusters within clusters → Selected leaf data (Reach the data to inspect)

### Full-text search must fetch, combine and score enormous lists

[17:19](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1039s)

Eskildsen closes the storage explanation with a simplified model of full-text search: a map from each token to the document IDs containing it. For `New York population`, the engine looks up three entries and combines their document-ID sets. It also ranks candidates. A rarer term such as `York` can carry more ranking information than a common term such as `New`; term rarity is the aspect of BM25 that he uses to explain the scoring.

### Two costs: storage waits and local list processing

The engine first fetches the relevant portions of the term dictionary, possibly after an additional lookup to locate them. It then fetches the document-ID lists, also called posting lists. The stages limit how much work can happen before the previous information arrives. Once those lists reach the node, combining and scoring them creates another cost: reading and processing large amounts of data through memory.

- **Compress the lists:** Smaller representations reduce the amount of posting-list data fetched from storage.

- **Reduce dependent requests:** Organizing dictionary and list access into a few rounds limits remote-storage waits.

- **Avoid weak candidates:** Once the engine has enough high-scoring documents matching `population` and `York`, documents matching only `New` may no longer affect the result. Scoring can therefore help avoid unnecessary list processing.

![Selected presentation frame from Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora at 1110 seconds](https://ai.engineer/api/talk-essay-media/01e44215403d3db05a3a626a47b6911bbe51153f78e3a9638fbb82e17a33a912)

Full-text search still has to move enormous lists

The last example gives the intuition for pruning work, without specifying the algorithm used to prove that a candidate cannot enter the results. It also makes Eskildsen’s closing comparison less surprising: he says full-text search at web scale is more difficult and computationally expensive than vector search. A familiar keyword query can involve enormous document-ID lists, storage transfers and memory-bandwidth demands.

### The product payoff depends on the workload fit

Lauritzen returns to why this matters for Legora. Retrieval is central to legal reasoning, but running a separate Elasticsearch database for every enterprise tenant would create substantial operational work. Namespace-level storage and encryption let the team accommodate residency and customer-key requirements without multiplying independent search deployments in the same way.

The cost advantage follows a particular shape: many independent collections, a few active ones, and a long tail of cold indexes whose occasional fetch delay is acceptable. Slower ingestion is also a reasonable tradeoff for these search workloads. Projects and jurisdictions give the engine useful units to cache, store and encrypt independently.

Lauritzen ends with the engineering consequence: the team can spend more time improving Legora and less time working on scalability and infrastructure. Its CFO is pleased with the cost, too. Making storage follow how legal work actually arrives and goes idle gave the team room to focus on the application.

## Key takeaways

- Roughly 4,000 partitions still mixed active and inactive projects. One namespace per project let cold collections stay in object storage without sharing a large partition with active work.
- Direct object-storage writes accept hundreds of milliseconds of latency. Responsive reads require indexes and query plans that do substantial work within a few storage round trips.
- Customer encryption requirements can include SSD caches. Disabling the disk cache performed well enough for some Legora workloads, avoiding the need to implement encrypted caching first.
- Legal research fans out because jurisdictional hierarchy, temporal validity and regulatory exceptions require additional retrieval. The application must resolve those relationships beyond finding similar passages.
- Cluster trees let frequently consulted vector-routing information stay in DRAM while larger leaf data occupies cheaper tiers. Full-text search also needs to control posting-list transfers and memory-bandwidth costs.
- The economic fit depends on a long tail of cold collections and tolerance for occasional cold-read delays. For Legora, that fit reduced infrastructure work and freed engineering time for the product.

## Resources

- [turbopuffer: fast search on object storage](https://turbopuffer.com/blog/turbopuffer) — Eskildsen’s companion explanation of object-storage-first search, cache locality and the tradeoff between data fetched per round and additional storage requests.
- [Napkin Math](https://github.com/sirupsen/napkin-math) — Performance numbers and estimation techniques for reasoning about storage latency, throughput and memory costs. Its rounded figures are guides for estimation, rather than guarantees for a workload.

## Related talks

- [Agents need more than a chat](https://ai.engineer/talks/XNtkiQJ49Ps-agents-need-more-than-chat) — Extends the legal-work discussion from retrieval infrastructure to collaboration interfaces, task verifiability and controllable human-agent workflows.

## Slides

### How to Connect AI to Billions of Legal Documents

Jacob of Legora and Simon of turbopuffer introduce their talk on connecting AI to legal documents.

[0:29](https://www.youtube.com/watch?v=V-isu4eTHgw&t=29s) [Title slide for a talk about connecting AI to legal documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/6443ae35836bb43593d9e6e3e334cc6186ee3ec03498ea01d685a3399e4ab9fe.webp)

How to Connect AI to Billions of Legal Documents

AI ENGINEER WORLD'S FAIR
2026-06-30

### Legora’s Collaborative AI Platform

Legora brings contract review and drafting, legal research, and collaboration into one platform for law firms and in-house legal teams.

[0:38](https://www.youtube.com/watch?v=V-isu4eTHgw&t=38s) [Platform overview showing review and drafting, legal research, and collaboration interfaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c33dc8b5cb532b75a7156332649c5981f9ebd11ffaa95f370c22d4998f47dc9.webp)

Legora is the collaborative AI platform for exceptional lawyers

REVIEW & DRAFTING
LEGAL RESEARCH
COLLABORATION

Legal research example:
Based on current jurisprudence, the delegation of special powers to the executive remains subject to strict constitutional limits regarding duration, necessity, and proportionality.

Constitutional Limits on Executive Emergency Powers

Under the Belgian legal framework, the use of special powers (bijzondere machten) allows the executive branch to issue decrees with the force of law during periods of national crisis. However, as established in various advisory opinions by the Council of State (Raad van State), these powers are not absolute. Each measure must strictly adhere to the principle of proportionality, ensuring that any restriction of individual rights is no greater than necessary to achieve the specific public health or safety objectives defined in the enabling legislation. Failure to provide a detailed justification for the urgency or the scope of the measure often leads to a recommendation of annulment.

Furthermore, recent jurisprudence emphasizes that even when the legislature grants broad discretionary power to the government, the Constitutional Court retains the authority to review these acts for compliance with the Division of Powers. This ensures that the executive does not encroach upon the essential functions of the legislative branch.

7 Sources

Interface text:
LEGORA
COLLABORATIVE AI FOR LAWYERS

Thinking
Edits 7 suggestions
1. Obligations - European alignment
2. Return/Destruction - GDPR
3. Feedback - moral rights
4. Injunctive relief - European
5. Mutual non-solicitation
6. Data protection clause
7. Miscellaneous - balanced
Dismiss all
Apply 14 edits

Worked for 6s
Citations 9
Results 9
No child left behind act of 2001
[illegible]
No child left behind act of 2001
[illegible]
No child left behind act of 2001
[illegible]

Private / Acme's Portal Intake
Share
Add item
Name | Due date | Parties | Assignee
Review amended Terms & Conditions | Mar 12,... | Buyer | [illegible]
Create mutual NDA — Project BrightWave | Mar 15,... | Buyer | [illegible]
Draft employment contract for new hire | Mar 18,... | Buyer | [illegible]
Flag risks in term sheet | Mar 20,... | Buyer | [illegible]
Send us the SPA issues list | Mar 22,... | Buyer | [illegible]
Advise on non-compete clause... | Mar 25,... | Buyer | [illegible]
Proofread e-mail to counterparty | Mar 27, 2026 | Buyer | [illegible]
Review IP license agreement | Mar 29,... | Buyer | [illegible]
Advise on GDPR compliance gaps | Mar 30,... | Buyer | [illegible]
Prepare board resolution template | Mar 31,... | Buyer | [illegible]

### Legora Growth Overview

Legora reports 1000+ customers across 50+ countries, $855m raised from top investors, and 600+ employees from top-tier law firms, big tech, and consultancies.

[1:16](https://www.youtube.com/watch?v=V-isu4eTHgw&t=76s) [Company overview showing customer, funding, and employee metrics alongside organization logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8389965a689fcfdb4ea1ff436f1a57a07c0252bddd7fbbfc3b109ae59fbeac5.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

One of the world's fastest growing enterprise B2B companies

1000+
Customers across 50+ countries

GPBL
GIANNI ORIGONI &
WHITE & CASE
GHELLA 1894
audax renewables
VATTENFALL
BARCLAYS
STRABAG
Bird & Bird

$855m
Raised from top investors

Accel
Bessemer Venture Partners
BENCHMARK
Redpoint.
ICONIQ
GENERAL CATALYST

600+
Employees from top tier law firms, big tech, and consultancies

Cooley
Baker McKenzie.
BCG
K
G
LW
WHITE & CASE
McKinsey & Company
Goldman Sachs

### Search at Legora

Legora searches documents within a project, from tens to millions of files. Legal research searches laws, regulations and past cases to support legal questions and litigation.

[1:24](https://www.youtube.com/watch?v=V-isu4eTHgw&t=84s) [Two-column comparison of project document search and legal research, with example questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cccd727bc82f5222b7f85b66cedda306a50162a66741abba36dabbaa2f434fe.webp)

Search at Legora

Projects
Search files in a project (10s-Ms documents)

Examples:
• Due diligence: What are the red flags in an acquisition? Look across all contracts and agreements
• Litigation: I want to sue someone (or I'm getting sued), find evidence

Legal Research
Deep research across laws, regulations, historic cases

Examples:
• How do I handle tax on income earned across three different EU countries?
• Can we enforce a non-compete on an employee who's moving to another country?
• I'm getting sued, help me find legal support that what I'm doing is ok

LEGORA
COLLABORATIVE AI FOR LAWYERS

### Search Over Documents

As project search grew from hundreds of thousands to two billion documents, Legora moved from single-region Elastic through multi-region Elastic and multi-tenant Postgres to turbopuffer.

[2:40](https://www.youtube.com/watch?v=V-isu4eTHgw&t=160s) [Timeline of project-search infrastructure from a single-region Elastic deployment to turbopuffer, alongside a rising document-volume curve.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9a0c53d45e5451cea2133a222a5d2736a2bb04d9ac66de6799ea479cda967c1.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

SEARCH APPLICATION #1
Search over documents

JAN 2024
Elastic single-region
SIMPLE START

NOV 2024
Elastic multi-region
MULTIPLE DEPLOYMENTS

MAY 2025
Postgres multi-tenant
CMEK, FULL ISOLATION

NOV 2025
turbopuffer
CMEK, FULL ISOLATION
HIGH PERFORMANCE

Y-axis ticks: 1M, 1B

### Elastic single-region, single-tenant

Document volume climbed increasingly steeply from January to November 2024. At the time, Legora used a single Elasticsearch cluster for its search workloads.

[2:51](https://www.youtube.com/watch?v=V-isu4eTHgw&t=171s) [Chart showing document growth from January to November 2024.](https://assets.ai.engineer/sites/aiecode2025/sha256/667d4ae3ba2f6b1a6864079849ec57f68288b7457267a58ffbd4aa32bdbccf91.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

01 Elastic single-region, single-tenant

Y-axis: M documents
Y-axis ticks: 0, 1, 2, 3, 4
Timeline: Jan 2024, Feb 2024, Mar 2024, Apr 2024, May 2024, Jun 2024, Jul 2024, Aug 2024, Sep 2024, Oct 2024, Nov 2024

### Single-region search architecture

Documents enter shared blob storage and Elasticsearch, with one index per tenant. Queries combine BM25 and vector search, then rerank the results.

[2:59](https://www.youtube.com/watch?v=V-isu4eTHgw&t=179s) [Diagram of document ingestion, tenant storage, and search, with one index per tenant.](https://assets.ai.engineer/sites/aiecode2025/sha256/7b7be0bfde0af58f41ceca4fc435e0c3f6cc1aeb77c8bfe0ae5f0aa4a83430f8.webp)

Elastic single-region, single-tenant

Setup
One index per tenant

Advantages
Simple

Querying
Expand query
Embed queries
BM25 search
Vector search
Reranker
Results

Blob
Tenant 1
Tenant 2
Tenant 3
Tenant 4

Elasticsearch
Tenant 1
Tenant 2
Tenant 3
Tenant 4

Ingesting
Store document
Parse document
Embed chunks
Index chunks

SCALING INFRASTRUCTURE
ENGINEERING CONFERENCE 2026

### Full processing isolation per region

Entering the US brought a requirement for full processing isolation by region. The speaker says Americans, Europeans and Australians wanted processing to stay within their respective regions.

[3:18](https://www.youtube.com/watch?v=V-isu4eTHgw&t=198s) [Regional processing isolation requirement illustrated with an eagle and a US flag.](https://assets.ai.engineer/sites/aiecode2025/sha256/079f1a06dd5c58a2ef1419211806d42bf5a9efe6de70f5bd76c76e9422971051.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

NOV 2024

Requirement: Full processing isolation per region

Adobe Stock | [illegible]

### Elastic multi-region, single tenant per region

The document count rises from near zero in January 2024 to over 40 million by May 2025, accelerating sharply in 2025. The chart marks November 2024.

[3:33](https://www.youtube.com/watch?v=V-isu4eTHgw&t=213s) [Chart showing document growth through May 2025, with a marker at November 2024.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8f967643af25619af354ed0149153e083a13a3974e42ba7c5057423b6f015c7.webp)

02 Elastic multi-region, single tenant per region

LEGORA
COLLABORATIVE AI FOR LAWYERS

Y-axis: M documents
Y-axis ticks: 0, 10, 20, 30, 40, 50
Timeline: Jan 2024, Mar 2024, May 2024, Jul 2024, Sep 2024, Nov 2024, Jan 2025, Mar 2025, May 2025

### Multi-region search architecture

Legora replicated its full search and ingestion stack across EU, US, and APAC regions, keeping processing—including LLM inference—in-region. The tradeoff: operating three separate setups.

[3:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=216s) [Diagram of replicated regional search and ingestion infrastructure, with its advantages and operational overhead.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc2657f59c5382e4c0e707280b65bfba92a3aaa2891884c3139ee46923a293a7.webp)

SCALING INFRASTRUCTURE
ENGINEERING CONFERENCE 2026

Elastic multi-region, single tenant per region

Setup
Full compute and storage per region.

Advantages
Fully encapsulates all processing (incl. LLM inference) to a specific region

Downsides
Operational overhead of running 3 separate setups

for x in {EU, US, APAC}:

Querying
Expand query
Embed queries
BM25 search
Vector search
Reranker
Results

Blob
Tenant 1
Tenant 2
Tenant 3
Tenant 4

Elasticsearch
Tenant 1
Tenant 2
Tenant 3
Tenant 4

Ingesting
Store document
Parse document
Embed chunks
Index chunks

### Enterprise requirements

Large banks and law firms ask for full physical tenant isolation and customer-managed encryption keys. Legora uses each customer’s key to encrypt and decrypt data at rest; revoking access prevents decryption.

[3:56](https://www.youtube.com/watch?v=V-isu4eTHgw&t=236s) [Enterprise requirements: physical tenant isolation and customer-managed encryption keys.](https://assets.ai.engineer/sites/aiecode2025/sha256/7cbc220fb3ed909a98269b9133b02b20d47df40284dd0e217b66f7dd3b9608d8.webp)

Enterprise requirements

MAY 2025

1. Full physical tenant isolation
2. Customer-managed encryption keys

LEGORA
COLLABORATIVE AI FOR LAWYERS

### Postgres multi-tenant

Document growth accelerates sharply through 2025. Legora moved search from Elasticsearch to Postgres, which it already used for OLTP.

[4:46](https://www.youtube.com/watch?v=V-isu4eTHgw&t=286s) [Chart of document growth from January 2024 through November 2025, rising to approximately 280 million documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ebd4a92770168a7f527d45569e26dcfb06d5941f3cfd4628d8aba5901535872.webp)

03 Postgres multi-tenant

LEGORA
COLLABORATIVE AI FOR LAWYERS

Y-axis: M documents
Y-axis ticks: 0, 100, 200, 300
Timeline: Jan 2024, Mar 2024, May 2024, Jul 2024, Sep 2024, Nov 2024, Jan 2025, Mar 2025, May 2025, Jul 2025, Sep 2025, Nov 2025

### PG multi-tenant

Legora moved search into its existing Postgres setup, hashing projects across partitions. It gained one operational system and isolation, but large, uneven partitions and tsvector search brought cost and retrieval tradeoffs.

[4:56](https://www.youtube.com/watch?v=V-isu4eTHgw&t=296s) [Postgres multi-tenant setup, advantages and downsides alongside a querying and ingestion architecture diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3bdf219d174ec2efeaf88d64d0327710b27c4c25e412e4b4de3c09df49e4f4f.webp)

PG multi-tenant

Setup
pgvector (diskANN) and tsvector
Hash-partitioned tables (hash key = project)

Advantages
One operational world (already using PG for OLTP)
Isolation “for free”

Downsides
• Unbalanced, large partitions put PG under heavy load for simple queries
  • Hot and cold projects on same partitions
• High operational + $ cost
• tsvector, not BM25

Querying
Expand query
Embed queries
BM25 search
Vector search
Reranker
Results

Tenant 1 blob
Tenant 2 blob
Tenant 3 blob
Tenant 4 blob

Tenant 1 PG
Tenant 2 PG
Tenant 3 PG
Tenant 4 PG

Ingesting
Store document
Parse document
Embed chunks
Index chunks

SCALING INFRASTRUCTURE
ENGINEERING CONFERENCE 2026

### With more data, PG died

DiskANN indexes were 4x the underlying data size. As partition indexes outgrew memory, queries thrashed the cache, and p99 retrieval and insert latency rose from hundreds of milliseconds to about 20 seconds.

[5:46](https://www.youtube.com/watch?v=V-isu4eTHgw&t=346s) [DiskANN index size and memory pressure are linked to approximately 20-second p99 retrieval and insertion latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/95c6b5f678141a117e9fbe63d8b089bdad44dd1d055f28e4530098dbf8c67a44.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

NOV 2025

With more data, PG died

DiskANN indexes 4x the size of underlying data
The working set across partition indexes couldn’t fit in memory

=> ~20s p99 retrieval and insert latency (initially 100s of ms)

### turbopuffer

Legora moved to turbopuffer at roughly 400 million documents; the chart shows document volume accelerating into 2026.

[6:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=396s) [Chart shows document growth from January 2024 into 2026, reaching roughly 1.7 billion documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/333838063e1f2b30eabd8b4495dde342a50826aa3b2e9b5fa7a2808ed02cc9bc.webp)

04 turbopuffer

Y-axis: M documents
Y-axis ticks: 0, 500, 1000, 1500, 2000
Timeline: Jan 2024, Apr 2024, Jul 2024, Oct 2024, Jan 2025, Apr 2025, Jul 2025, Oct 2025, Jan 2026, Apr 2026

LEGORA
COLLABORATIVE AI FOR LAWYERS

### turbopuffer

Legora uses one namespace per project in a single cluster. Queries combine BM25 and vector search before reranking; inactive projects stay in blob storage, reducing cost and operating complexity.

[6:44](https://www.youtube.com/watch?v=V-isu4eTHgw&t=404s) [One namespace per project, retrieval and latency improvements, and a single-cluster architecture for querying and ingestion.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcc305347f5bc551678035706980e1af4c126ad98966bdba8e5d30d150bb2dd2.webp)

SCALING INFRASTRUCTURE
ENGINEERING CONFERENCE 2026

turbopuffer

Setup
One namespace per project

Advantages
Improved retrieval performance (proper BM25)
Improved latency

A single cluster across all tenants with native isolation (cold projects stay cold)
=> Much lower cost
=> Way simpler to operate

Querying
Expand query
Embed queries
BM25 search
Vector search
Reranker
Results

Tenant 1 blob
Tenant 2 blob
Tenant 3 blob
Tenant 4 blob

Ingesting
Store document
Parse document
Embed chunks
Index chunks

### Object-storage-native search engine

Writes land in an object-storage write-ahead log while indexers build indexes in the background. Query nodes check memory and NVMe caches before object storage.

[7:29](https://www.youtube.com/watch?v=V-isu4eTHgw&t=449s) [Diagram of indexers and query nodes with memory and NVMe caches, backed by an object-storage indexing queue, write-ahead log and index.](https://assets.ai.engineer/sites/aiecode2025/sha256/0fe39818dea04a958eee63b81ebe90e72bf249d4b82a4fe1b6dc37e57c737101.webp)

Object-storage-native search engine

Client → LB

turbopuffer Region
./tpuf indexer
./tpuf indexer

./tpuf query
Memory Cache
NVMe Cache

./tpuf query
Memory Cache
NVMe Cache

Object Storage
Indexing Queue
/{org_id}/{namespace}
/wal
/index

* unlimited read replicas
* query-index separation

### Latencies improved by an order of magnitude

The dashboard compares postgres with turbopuffer: inserts are 14.5 s vs 294 ms, semantic searches 1.34 s vs 80.9 ms. The panels say average; the speaker calls them medians and says savings add up across agent queries.

[11:40](https://www.youtube.com/watch?v=V-isu4eTHgw&t=700s) [Dashboard compares PostgreSQL and turbopuffer insert, semantic search, sparse search, and chunk retrieval latencies.](https://assets.ai.engineer/sites/aiecode2025/sha256/be0fefda96cbfdf52a614ca8aa5d2c9d8306854973999fc5ddc939059b8781d8.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

Latencies improved by an order of magnitude

KPIs

Average Insert Latency
postgres 14.5 s
turbopuffer 294 ms

Average Semantic Search Latency
postgres 1.34 s
turbopuffer 80.9 ms

Average Sparse Search Latency
postgres 817 ms
turbopuffer 78.2 ms

Average Get Chunks Latency
postgres 255 ms
turbopuffer 105 ms

Interface text:
Datasource: prod-eu
All
now-10d to now-7d
Refresh 30s

### Legal Research

The chart rises slowly through 2025, then steeply toward a Q1 2027 projection. The quantity being charted is not labeled.

[12:12](https://www.youtube.com/watch?v=V-isu4eTHgw&t=732s) [Area chart rises steeply from 2024 to a projection in Q1 2027.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4ee105a9ea86589d8b20ed9eed5e6019d8f516a90ae4460b9ede88457f79827.webp)

LEGORA
COLLABORATIVE AI FOR LAWYERS

SEARCH APPLICATION #2
Legal Research

Y-axis ticks: 1B, 4B
Timeline: Q1 2024, Q3 2024, Q1 2025, Q3 2025, Q1 2026, Q3 2026, Projection Q1 2027

### What makes it hard?

Legal research fans queries out to account for legal hierarchy, overruled decisions and exceptions, across 10B+ vectors and 1B queries per month.

[12:17](https://www.youtube.com/watch?v=V-isu4eTHgw&t=737s) [Bullets describe vector scale, query volume, and hierarchical, temporal, and conditional filtering.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f3c7fa6f189ec21aec9692343f72391d6f78c2ea2acc9d4295f1fb6fd4727b9.webp)

What makes it hard?

• Huge scale: 10B+ vectors and growing quickly
• High QPS: 1B queries/mo, spikes to 1k QPS
• Heavy filtering for graph-like retrieval:
  • Hierarchical (statutes > federal case law > agency decision etc)
  • Temporal validity (are there other documents that explicitly or implicitly overrule this one?)
  • Conditional connections (are there any exemptions, pre-conditions etc to these rules that are stated in other docs)

Interface text:
LEGORA
COLLABORATIVE AI FOR LAWYERS

## Full transcript

[0:12](https://www.youtube.com/watch?v=V-isu4eTHgw&t=12s) Go.

[0:13](https://www.youtube.com/watch?v=V-isu4eTHgw&t=13s) Okay. Hi everyone. Welcome, um, this 20-minute talk about connecting AI to loads of legal documents. My name's Jacob. I'm an engineer at Legora.

[0:26](https://www.youtube.com/watch?v=V-isu4eTHgw&t=26s) Yeah, and I gotta step into frame here. Uh, I'm Simon. I'm the CEO and, uh, CEO and co-founder of TurboPuffer, a search engine that we work with Legora and others on.

[0:35](https://www.youtube.com/watch?v=V-isu4eTHgw&t=35s) It, um, super quickly, um, introduction to Legora. We're a collaborative AI platform for legal work, and so that means we have law firms that are clients, and we have, uh, in-house legal teams that are clients, and they use Legora to do reviews of contracts. They use it to go through an absurd amount of contracts and make sure that they all look good. They use them to, uh, create new contracts. They do legal research, which means, um, looking over all, uh, potential law,

[1:06](https://www.youtube.com/watch?v=V-isu4eTHgw&t=66s) and they collaborate inside Legora. So you can think of Legora sort of as a Linear/Figma/Notion/GitHub for legal work. It's a lot. We, um, are one of the fastest growing companies, uh, right now. We grow extremely fast. Yeah, tons of numbers on the screen. I'll just skip through that. What we really wanna talk about is search today. So at Legora, there's two types of search that we do. There is project search and legal research.  Project

[1:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=96s) search is, um, basically projects in Legora is, like, the, the unit of work that you have. So if, um, let's say you are SpaceX and you want to acquire Cursor, then that would be one project with your law firm. Uh, and so they would go into Legora, and they would upload all these documents, and the law firm that helped you would go through all of the employment agreements, all of the contracts with suppliers. Um, I know Cursor is using, uh, TurboPuffer, so maybe there's a contract there they'd look at. Um, but

[2:05](https://www.youtube.com/watch?v=V-isu4eTHgw&t=125s) basically you do the search confined to a project, and projects can be tens of documents to millions of documents. The other use case is legal research, and legal research is sort of a deep research-style workload where we'll search across tons of laws, previous cases, regulations, et cetera, et cetera, and people use this to answer questions such as, like, "How do I handle this specific thing?" And they'll also use it for litigation. Maybe they want to sue someone or maybe they are getting sued, and they'll use legal research to support and help their case.

[2:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=156s) So if we start at number one, project search. We've been through a little bit of a ride here on how we do search, um, starting at, you know, hundreds of thousands of documents all the way into two billions of documents, and we've tried a lot of different things. So first we start, started with a very, very simple one, which is just a single Elasticsearch cluster for all of our search workloads. Um, that worked relatively well. It was sort of a simple setup. All of the tenants, you know, our clients, our users,

[3:07](https://www.youtube.com/watch?v=V-isu4eTHgw&t=187s) would be on one big blob storage, where we store the raw documents, and on one big Elasticsearch, where we would do all of their searching, the indexing and the searching. Super simple. Worked relatively well initially. Then we wanted to enter the land of the free, and, uh, we got some new requirements. They... You know, Americans only want processing to happen within the US, and Europeans only want it to happen within the EU, and Australians only want it to happen within Australia. And so we had to s- basically, um... Well, here's a scaling

[3:36](https://www.youtube.com/watch?v=V-isu4eTHgw&t=216s) graph. We had to move to multiple Elasticsearches, and what we actually did was we took the entire setup, and we just basically iterated over the set that is EU, US, and Asia Pacific, and so we just had this, like, multiplied by three or four. Kind of annoying, lots of overhead, but it, uh, got us to where we needed to be.

[3:57](https://www.youtube.com/watch?v=V-isu4eTHgw&t=237s) Then the next iteration of the story is enterprise. So really big banks, the biggest law firms in the world, they have really annoying requirements. And number one they have is they'll ask for full physical isolation of all of their data. There's probably a little bit of a war on what physical  you know, isolation actually means, but essentially it means they want their own database. They also want customer-managed encryption keys, and what that means is they basically have a key vault thing where they have an encryption key, and they give us

[4:27](https://www.youtube.com/watch?v=V-isu4eTHgw&t=267s) access to read the key, and we then use that key to, uh, encrypt and decrypt all of their data at rest. And what that gives them is they can just revoke our access to their key, and then we can't decrypt their data anymore, and so it's safe. And so in a way, that gives enterprises a lot of control over all of their data 'cause they control, you know, the key to, to reading it.

[4:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=288s) So we moved from Elasticsearch to Postgres, and I imagine a bunch of you guys are like, "Why would you ever put your vectors into Postgres?"  Um, it actually works surprisingly well, and the reason that we did this was we were already using Pro- Postgres for OLTP workloads. And, uh, so we, we sort of already had to do this split of multiple Postgreses and multiple blobs. And so it was really easy for us to try to shift all of our search into Postgres as well 'cause then we only have one system. So the setup here was pgvector, uh,

[5:18](https://www.youtube.com/watch?v=V-isu4eTHgw&t=318s) specifically diskANN, tsvector for the search, so not BM25, which was, you know, we lost a little bit of, uh, retrieval performance there. And what we'd do is we would partition the table where we would store all the document chunks. We'd partition it super aggressively, like 4,000 partitions, and then each project, we'd basically hash the project key, and we'd bin pack them into the partitions. That actually worked relatively well, but it was expensive, and search performance wasn't super, super good. And what happened was when we scaled

[5:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=348s) a lot, uh, everything just broke and exploded. And so what happened was the... Basically, you can imagine, like, you have a bunch of projects, and some of them you spin up a project, you work on it, and then you close it, and you basically never go back to it again. And we have a bunch of those where, like, they never get queried, and we have a bunch that get queried all the time 'cause they're super active projects. And when we pack them into partitions, the cold ones and the hot ones would land on the same ones, and the partitions would get really, really big. And so when we queried them, they would... Postgres would pull the partition, put it into memory, we'd do the

[6:17](https://www.youtube.com/watch?v=V-isu4eTHgw&t=377s) stuff, and then we'd query another partition, and another partition, and it would essentially thrash the cache all the time. And what that meant was our latencies would spike. So we went from, like, search and ingestion P99 of a hundred milliseconds into twenty seconds, which you can imagine is a really bad user experience.

[6:37](https://www.youtube.com/watch?v=V-isu4eTHgw&t=397s) So then we went to Turbopuffer in about, you know, when we were about, I think, four hundred million documents, something like that.

[6:45](https://www.youtube.com/watch?v=V-isu4eTHgw&t=405s) And what we did with Turbopuffer was we did one namespace per project. And the advantages of moving to Turbopuffer is we got BM25, real BM25, much better, uh, relevancy, much better latencies, and it was extremely simple to operate 'cause we could just have a single Turbopuffer cluster. You know, we didn't have to have a bunch of different ones like with Postgres, and it would, since it's blob-based, it could just query the blobs that we had anyway. And so much lower cost, and it was extremely simple to operate,

[7:16](https://www.youtube.com/watch?v=V-isu4eTHgw&t=436s) and we didn't have this problem with the partitions 'cause if a project's not used, it's just in blob. And so it's really easy. And Simon can talk a bit more about why that works so well.

[7:28](https://www.youtube.com/watch?v=V-isu4eTHgw&t=448s) Yeah. So Legora has some of... And, and legal in general has... By the way, if, if, uh, Jake and I s- have similar accents and maybe even look a bit similar, it's because we're both Danish. Um, Turbopuffer has a very, has a particular architecture that supports these kinds of very regulated environments really, really well. But in order to understand that, we have to understand what kind of search engine is Turbopuffer. Why is it different than the ones that they used in the past? Since the very beginning of Turbopuffer, um, the design has more or less been the same. Uh,

[7:58](https://www.youtube.com/watch?v=V-isu4eTHgw&t=478s) there may be changes in the future, but the design has stood the test of time. When you do a write to Turbopuffer, we write directly to object storage. There is no, like, disk replication, there's no Paxos, there's, there's none of that. Direct to S3. That's the fundamental trade-off in Turbopuffer, right? Hundreds of milliseconds. If you're like Shopify and doing inventory, uh, reservations for a Kylie Jenner flash sale, not gonna work. Very, very good for search because generally when you're doing search, doing a slow

[8:28](https://www.youtube.com/watch?v=V-isu4eTHgw&t=508s) write is fine, um, as long as the read performance is, is adaptable and good. So that's what happens on write. It just goes in the write-ahead log. You can imagine you write one.json, two.json, three.json. Obviously, it's a da- it's, it's a database, so it's not JSON, but for illustrative purposes, that's what happens. And in the background, we build the vector indexes, the text indexes, the columnar indexes, and so on to satisfy the queries that, that Jake and other customers have. Um, so then at query time, we can go in and then, um, the query

[8:58](https://www.youtube.com/watch?v=V-isu4eTHgw&t=538s) reaches some namespace, and namespace is kind of our concept of a, of a table. Um, you can think of it as a directory on S3 that's isolated from everything else. We go to the node that is most likely to have it. It could go to any node, right? It could go to every single node, and they're all read replicas, but it would... we go with some affinity to the node that has the highest probability of having it in cache. We check the memory cache for any objects, NVMe SSD cache, and then finally to object storage. Everything in Turbopuffer is optimized around doing as much work in as few round trips as possible, right? S3 has a P99,

[9:28](https://www.youtube.com/watch?v=V-isu4eTHgw&t=568s) um, on a, like, one megabyte blob size of around, uh, two hundred milliseconds, so you wanna do as few round trips as possible, right? Ideally, you do around three. And everything in Turbopuffer, the database, is... Oh, I'm gonna need your fingerprint.

[9:41](https://www.youtube.com/watch?v=V-isu4eTHgw&t=581s) You got it.

[9:42](https://www.youtube.com/watch?v=V-isu4eTHgw&t=582s) Um, everything in, in Turbopuffer is designed around minimizing the number of round trips. This is also amazing for modern dri- disks. If you do a lot of concurrency in few round trips, you utilize them optimally, and everything in Turbopuffer is designed around this. So why is this so good for a company like Legora? Well, object storage native, if you design it around the atomic unit of separation being the namespace or the table, every single table could be encrypted with a different key. Every single namespace could be in a different bucket. We have

[10:12](https://www.youtube.com/watch?v=V-isu4eTHgw&t=612s) customers that have, um, thousands of buckets that they have namespaces in so that their customers get the warm IT fuzzies of having the bucket in their own cloud account. They can also be encrypted with their own keys. You can share buckets. You can, you can do whatever configuration that you need at the namespace level. You can re-encrypt with different keys, you can move them around, um, and you can re-encrypt with other keys. Um, for Legora in particular, this was really important for this full physical separation, right? An encryption

[10:41](https://www.youtube.com/watch?v=V-isu4eTHgw&t=641s) separation. All of the namespaces needed to be physically at rest with different keys and as separate as possible. S3, GCS, Azure Blob Storage, they pass that. Um, and the other parts of the hierarchy also. Except the NVMe SSD cache, because in the SSD cache, we consider that to be volatile like memory, um, but your customers did not.  So in the, uh... So what we did was that we thought we were gonna implement encryption into the disk cache, but instead we just disabled the disk cache

[11:11](https://www.youtube.com/watch?v=V-isu4eTHgw&t=671s) and saw how it fared. And the performance of Turbopuffer, even without the disk cache with just the memory cache, was so good that we just kept it that way for some of the Legora workloads where we couldn't have the disk cache for multi-tenancy. Turbopuffer will support that in the future. But it just goes to show the, um, natural point where Turbopuffer allows these encryption and storage and separation to become fully multi-tenancy, uh, native. I'll hand it back to you on what happened then.

[11:39](https://www.youtube.com/watch?v=V-isu4eTHgw&t=699s) And then, drum roll please, latencies looked like this. Um, is my mic working?

[11:47](https://www.youtube.com/watch?v=V-isu4eTHgw&t=707s) No.

[11:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=708s) No? Could I... Or I'll start screaming really loudly. Um, it speaks for itself if you can't hear me. Okay. Um, latencies improved in order of magnitude basically, and these are median latencies, so P99 were even better. Um, so obviously this is a huge thing when you're doing-- I mean, one thing is if you're doing a single sort of RAG-style thing, but if you have an agent that does twenty queries, a hundred queries, these really, really add up. So that was on the project side, and then a more recent thing is legal research.

[12:18](https://www.youtube.com/watch?v=V-isu4eTHgw&t=738s) So legal research, um, is a kind of a difficult problem, and the reason it's difficult is that, um, the corpus is extremely big, so we're racing towards ten billion vectors, and we're growing extremely fast. We also have quite high read, um, so QPS can spike a lot 'cause we do a lot of fan-out. Like, if you do a, a, a sort of legal research query, we will fan it out into a bunch of different queries, and we'll keep going. And the reason we do that is we need this, um, heavy filtering 'cause essentially

[12:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=768s) it's a-- kinda like a graph for a few different reasons. Firstly, it's, um, hierarchical. You know, you have cities, and you have counties, and you have states, and you have federal law, and it's the same all around the world. Um, and so you need to respect that authoritative sort of hierarchy. There's also some temporal validity, so one judge might overrule a decision that's been made somewhere else, and you need to also respect that and figure that out. And then sometimes there's even, like, a new regulation that has exemptions or special cases of an old regulation, and

[13:18](https://www.youtube.com/watch?v=V-isu4eTHgw&t=798s) so if you're finding this one, you need to find all the other ones as well. So you can imagine that it, uh, sort of explodes the search.

[13:26](https://www.youtube.com/watch?v=V-isu4eTHgw&t=806s) And so we started on Elasticsearch for this, but also moving to TurboPuffer. Um, Elasticsearch just got extremely expensive 'cause we have to have everything there. But, um, with TurboPuffer, we can basically, uh, take different jurisdictions, and we can make them namespaces in TurboPuffer, and that means some of them... Here's an example where, like, you have the EU. That gets queried all the time. That's super hot. And some of them, let's say Danish law 'cause we're Danish, no one cares really. It's such a small country, so, like, it doesn't really get queried, and so that can just stay on Blob, and that's fine.

[13:56](https://www.youtube.com/watch?v=V-isu4eTHgw&t=836s) Um, and because it's sort of a deep research-style workload, if there's five hundred milliseconds latency to fetch that cold Blob, that's okay. That's fine. It's not really a big problem. So the way that TurboPuffer is, is designed lends itself super well to this super long scale of, like, cold, weird namespaces and a few that are really, really hot. Um, yeah, and Simon wants to talk more about that.

[14:20](https://www.youtube.com/watch?v=V-isu4eTHgw&t=860s) Yeah. So, um, um, I was, I was talking about why the company is called TurboPuffer at another, uh, talk here earlier today. But, uh, one of the other explanations of the name of TurboPuffer is that it's about puffing into the different memory hierarchies and really mastering when data should be in particular memory hierarchies. So you can think about it here, right, of something like the EU law might be more or less part of almost every one of the legal research, uh, queries, right? So that probably sits closer

[14:50](https://www.youtube.com/watch?v=V-isu4eTHgw&t=890s) to NVMe SSDs than memory, right? The economics kinda change as you move up and down this hierarchy. Um, in, in memory, you want things that are queried a lot, right? Then the economics of memory are great. NVMe SSDs can-- you can do a lot of things directly on them, but the economics change as you move up and down this boundary. The latency changes, and the way that the database is architected to take advantage of it in terms of round trips versus random versus sequential all changes as you navigate this hierarchy. TurboPuffer is a database that is really designed around the memory

[15:20](https://www.youtube.com/watch?v=V-isu4eTHgw&t=920s) hierarchy, and all of the smarts in TurboPuffer is that all of these namespaces are puffed in and out, um, of the cache. You can think of this as we wanna spend as much time, have as much data pushed as far down in this hierarchy as possible to get the, the best, um, performance cost ratios. So how does that apply to search? Well, for something like vector search, for example, there's two fundamental ways to do vector search. One is to navigate it, basically design a graph. The problem with a graph on something like object storage or disk, again, we wanna

[15:49](https://www.youtube.com/watch?v=V-isu4eTHgw&t=949s) have things as far down that memory hierarchy as possible. The problem with a graph, this is not a graph, this is a tree, um, but in a graph, you have to navigate from the center of the graph. So-- And then every time you navigate through these nodes, you're doing two hundred millisecond P99 to S3, right? And so you're trying to shrink the diameter of the graph. You're trying to do all these tricks to make the graph. But fundamentally, you're at odds with the fact that a graph is about a random sequential trade-off that you have in memory and in registers but not further down the memory hierarchy.

[16:20](https://www.youtube.com/watch?v=V-isu4eTHgw&t=980s) The way TurboPuffer does it is organize it into clusters, right? Vectors you can think of in two dimensions just as point is a massive coordinate system, and we can organize them into clusters. TurboPuffer then creates clusters of clusters and clusters of clusters of clusters to essentially organize all of the vector data in a tree. You can basically think of TurboPuffer as a very, very complicated B-tree,  right? Because it's a tree on this geometry of this entire space and the clustering of it in an approximate way. Now, the root centroids further up

[16:50](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1010s) the tree, you can imagine, are part of every single time you search, right? They're-- We're always trying to figure out which clusters that we're in, and we're always looking at the upper levels of the tree. So they're gonna be further up the memory hierarchy, right? Closer to the registers, m-almost all in DRAM. Now, the leaves that have all of the actual legal cases of whatever long document it could be, it could be images, all of that is probably gonna be on SSDs with that single one-millisecond round trip at the end. It doesn't make sense to have all that puffed into DRAM. This is fundamentally the cheapest

[17:19](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1039s) way that you can run a database, period. So for something like Legora or even web search, which is in the hundreds of billion or tens of billions, depending on how much of the web you've scraped, this is fundamentally the cheapest way to do it. And we have customers that are indexing massive parts of the entire web into TurboPuffer, which is really also a part of what legal research is. Full text is also, also really respectful of the memory hierarchies. The way that text search works is essentially you can think of it as a HashMap. You have a

[17:50](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1070s) big document, and then you take every single one of the tokens, and you put them into the key in the HashMap. The value in the HashMap is some set with all of the document IDs that has that term. So then if you search for New York population, you're finding those three places in the HashMap, and then you're taking the three sets and doing an intersects on the- intersect on the sets. While you're intersecting, you're also trying to do some kind of scoring, right? A document that has York in it is probably more valuable than a document that has New in it because York is a more

[18:19](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1099s) rare word. When people say BM25, this is the scoring that they're referring to. The art of full-text search is, one, we wanna minimize the number of round trips. So first you download the parts of the dictionary that are relevant, round trip one, maybe a round trip one before that to index into where the parts of the term- terms are. And then the second round trip is to get these massive lists. Try to make the lists as small as possible by compressing them. But also while you're doing the text search, you're trying to minimize the amount, again, of memory bandwidth that you

[18:49](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1129s) want to intersect these lists. You can probably imagine that at some point, there's a point where you've seen so many documents with population and York that have much higher scores that documents that just have New in it are irrelevant anymore. This is like a mega crash course in how text search works. And counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search. I'll hand it over to you.

[19:16](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1156s) Cool. So key learnings from, um, what you heard today. Retrieval is extremely important to Legora. Um, it's key to legal reasoning. TurboPuffer really excels when, uh... for us 'cause, uh, it makes it extremely easy to operate. We have seventy-plus tenants. We have a hundred. We have two hundred tenants. You know, if we had to have separate Elasticsearch databases for each of these, it would be just hell. Um, but we can do this natively with TurboPuffer with, uh, data residency and CMAC, et cetera, et cetera.

[19:47](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1187s) And then it's extremely cost efficient generally when you have these types of workflows, um, or workloads that we do where there's a long tail of cold indices basically that you don't need to query so much, and you don't-- you, you're sort of... You're okay paying the small, uh, latency cost for it. So now with, um, TurboPuffer and four seconds to go, now we can focus on making Legora. We can focus on the product, making it really, really great, not on scalability and infra. And also David, our CFO, is really happy about the cost, so it's great. Thanks, everyone.
