If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread
Read the talk
If we want them to do Knowledge Work, design them as Knowledge Agents
Benjamin Clavié explains how search tools, delegated research, and concise memos help agents turn ambiguous information into useful judgments—and why a coding ticket hides much of that work.
From a talk by Benjamin Clavié
At a glance
Ideas worth remembering
Coding agents benefit from stable searchable names and tasks humans have already narrowed. Open-ended knowledge agents must also discover what needs researching.
Tune retrieval baselines and compare search effort alongside accuracy. Fewer tool calls can make a workflow practical, but call reductions alone do not establish total cost savings.
Multimodal retrieval exposes PDF evidence; delegated searchers organize findings into memos. The reported improvement narrows a gap to human performance with oracle evidence, rather than establishing a controlled comparison of research ability.
Co-design agents with their search primitives. Larger context windows still need task decomposition and deliberate selection of evidence.
From asking a PDF to producing a judgment
An early retrieval-augmented generation system could search a document and let someone ask questions about a PDF. It provided an interface for asking questions, but Clavié considers it limited as a worker. Benjamin Clavié, who works on retrieval at Mixedbread, begins with the next step: agents that search, interpret results, call tools, and carry out programming tasks. Their success creates a tempting assumption that the same design will work across other professions.
Legal research, financial analysis, and finding medication information all require an agent to find knowledge and use it. Coding belongs to that larger category, too. Clavié's concern is that agent designers have learned disproportionately from one unusually convenient form of knowledge work, then tried to generalize its tools and task structure.
His working definition starts with the transformation the worker performs: information goes in; an actionable judgment or decision comes out. The input can be ambiguous and scattered. A lawyer must turn it into findings about a case and perhaps an argument on the client's behalf. Finding documents supplies the material. Understanding their consequences produces the work.
Clavié also offers a deliberately circular shorthand: if the problem needs search, it is a knowledge problem; knowledge problems need search. Lawyers, academics, actuaries, software engineers, and researchers have developed ways to organize this work. Those established workflows give agent designers more examples to learn from than programming alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why thirty days is harder to search than a function
Code supplies both clues and a prepared task
Code gives an agent several advantages at once:
- Stable references: Identifiers, files, and paths often persist long enough to connect a request to its implementation.
- Searchable structure: Keywords and method definitions give
grepliteral strings to find. - Narrow assignments: A feature request or ticket usually arrives after a human has reduced a broader need to a particular change.
These advantages reinforce each other. A small task points toward a small part of a codebase, and that codebase contains names the agent can search.
Clavié makes the hidden preparation visible by imagining a much broader request: discover a new programming paradigm, then implement it in an application. Ordinary coding-agent requests rarely demand that kind of invention and decomposition. The human has already done some of the intellectual work before the agent starts searching.
A matching phrase can answer the wrong question
Now consider an agent searching a contract for thirty days. The phrase could describe a deadline, a grace period, or a retention rule. A search could even return a medication schedule from another domain. The words match in every case, but their role changes. The agent needs to establish what the duration governs and whether that meaning fits the question.
The search therefore starts from intent rather than a known identifier. A legal question might require finding international norms, identifying their conditions, and deciding whether those conditions hold in this case. The original request may name none of those dependencies. The agent must discover what it needs to know before it can retrieve enough evidence to answer. This is why Clavié treats non-code knowledge as especially contextual and meaning-driven.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Knowledge, tools, and specialized roles grow together
Clavié turns to a broad history of knowledge work to explain two developments:
- Tools for finding information: Writing, library catalogues, bibliographies, library classification, and search engines make growing collections easier to navigate.
- Organizations for dividing work: Reliance on one gifted expert gives way to institutions such as monasteries and universities, bureaucracies, and specialized firms.
The examples illustrate two responses to the same pressure: there is more knowledge than one person can conveniently find and use alone.
A hospital makes specialization concrete. Doctors, senior doctors, nurse practitioners, nurses, and healthcare assistants handle different levels of tasks. The organization improves the allocation of work alongside whatever improvements its tools provide. Expecting one expert to perform every step would discard that organizational advantage.
The two developments form one feedback loop. New knowledge demands better tools; better tools allow new workflows and roles. Workers must learn those tools before the new workflows become effective. Someone accustomed to visiting a library needs to understand that a search engine accepts a typed query without a physical visit. More efficient workers then create more knowledge, starting the cycle again.
Cheap search changes which questions are worth asking
A tool can make a task economically practical even when the task was already possible. Clavié imagines finding a manuscript in an uncatalogued library taking two or three weeks, compared with ten minutes using a catalogue. These are illustrative timings: their point is that expensive retrieval reserves information for urgent questions, while cheap retrieval lets people investigate smaller uncertainties.
Maps provide another example: information about where to go changes the practicality of exploration. Multimodal search makes the same argument for archives containing millions of PDFs. A collection people would otherwise leave untouched can support useful work once finding evidence costs less. Tool quality changes the set of workflows people are willing to adopt.
Growing information creates new retrieval needs.
Tools and organization improve together as knowledge grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tune the baseline, then count the work needed to answer
Clavié next compares retrieval tools on a bounded deep-research benchmark: specific questions over a large text collection. The leaderboard includes two BM25 configurations, optimized and unoptimized. BM25 performs lexical search—retrieval based on the words in queries and documents—but the method's name does not specify a single implementation or quality level. A poorly tuned baseline cannot tell you how much the method can accomplish.
The badly optimized tool reaches about 60% accuracy. Clavié considers that too unreliable for sustained use: if someone repeatedly needs to check or redo the work, doing it themselves becomes attractive. Better retrieval moves accuracy through the seventies and eighties toward 90%. Near the top, he treats small score differences as run variance, so another fraction of an accuracy point is less persuasive than a clear efficiency gain.
Similar accuracy can require much less searching
The result Clavié emphasizes is 90.2% accuracy with 20% fewer tool calls. Finding useful evidence sooner means fewer rounds of searching and processing results. He connects that to lower token use and resource costs, then contrasts a workflow requiring 25 calls with one requiring 8. Efficiency matters because the cheaper workflow is easier to keep using.
The cost claims need their conditions kept attached. Clavié also describes spending about 5% of the unoptimized baseline's cost, but the call counts alone do not establish that ratio. Nor does 20% fewer calls necessarily mean 20% lower total cost: calls and model turns can consume different amounts of work. The practical decision is to compare answer quality with the effort needed to achieve it, rather than choose a tool from accuracy alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A law firm separates framing, research, and synthesis
A bounded benchmark still understates the difficulty of an open-ended client problem. Clavié describes a client bringing a situation to a law-firm partner. The partner identifies the facts, likely consequences, and areas of law that need investigation. Before anyone searches, someone must convert the situation into researchable questions.
Paralegals and assistants investigate those questions using tools they have learned to use. They return memos and notes to the partner. The partner may research a clarification, but mostly relies on those findings to assemble the response. The memo is an important intermediate product: it gives the person responsible for the answer organized research to reason over.
This returns to the coding-ticket comparison. The programmer often performs the framing step before handing work to an agent. A client with a broad problem may provide only the situation and desired outcome. Clavié argues that a knowledge agent must therefore own the decomposition step and delegate research to subagents. Simply giving the answering agent a search tool leaves the organization of that work unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Better evidence removes one ceiling; memos reduce the remaining gap
When the tool limits even a human
A PDF-based enterprise benchmark provides both PDFs and OCR versions: text extracted from document images through optical character recognition. With BM25, Clavié reports that a human and a model reach the same performance ceiling, even when the human can continue searching. He interprets that equality as a retrieval limitation: the tool fails to expose the needed evidence. It does not establish that the model reasons as well as the human.
Multimodal retrieval changes what evidence is available. Mixedbread's search tool can work with the PDF's visual content, including tables, rather than depending entirely on OCR text. Clavié reports a large accuracy improvement from this change. The causal distinction matters: more thinking or more searches cannot reliably recover information that the representation and retrieval process fail to supply.
The better tool still leaves the agent at a reported 88.9% accuracy, compared with a 99.4% human result using oracle evidence—the correct documents supplied for each question. That condition matters: the human comparator is relieved of retrieval misses. It is different from the BM25-constrained human discussed earlier. Mixedbread's separate benchmark account identifies the 99.4% result as human performance with an oracle retriever and lists the human using BM25 at 82.2%; its other configurations should not be substituted for the recording's reported 88.9%.
The agent can inspect results and continue over multiple turns; Clavié recalls a budget of about ten turns. Better evidence and an iterative loop help, but the remaining difference from the human oracle-evidence result does not isolate reasoning ability. The agent still has to find and organize its evidence, while the oracle comparator already has the right documents.
Give the answering agent a research product
Mixedbread's search-agent approach applies the earlier law-firm pattern. The main agent breaks the question into important aspects and writes queries for them. Searcher agents investigate those aspects independently, find results, and return short memos. The main agent then answers from the organized findings. Research and synthesis have separate places in the workflow.
Clavié reports a 3.5-percentage-point accuracy gain. He describes its significance through the oracle gap: the distance between a search system's result and the result obtainable with ideal evidence. The useful question is how much performance retrieval and research organization leave unused, rather than how impressive the raw score looks.
Clavié rounds the gap from about ten points before searcher agents to about six afterward, describing roughly a 40% reduction. Using the recording's stated 88.9% agent score, 99.4% human oracle-evidence score, and 3.5-point gain gives a 10.5-point initial gap, with one-third closed by the gain. His rounded 40% description should therefore stay approximate. This is a reduction in the gap to an ideal-evidence comparator, rather than a controlled measure of human versus agent research ability. The experiment still shows additional improvement from organizing research after improving retrieval.
The answer may require several aspects of a large PDF collection.
Searcher agents gather evidence for the aspects identified by the main agent, then return organized findings for synthesis.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach search primitives and organize finite context
Clavié closes with a practical reason to study librarians, paralegals, legal firms, and medical organizations: humans have already developed ways to divide knowledge work. The benchmark examples show how to apply that lesson. Improve a tool when its limitations impose a ceiling; change the workflow when better evidence still leaves performance unused. Tools should serve an observed need.
An available tool is not automatically a usable tool
Tools and agents need to be designed together. Clavié observes that agents can default to grep-style queries because grep appears widely in training material. That familiarity does not make the query appropriate for every collection. A model or harness needs to understand the available search primitives:
- Grep: Find literal patterns in searchable text, including the stable names common in code.
- BM25: Perform lexical retrieval over an indexed text representation.
- Semantic search: Formulate a meaning-based query when literal clues do not adequately express the information need.
The agent must know when to use each primitive and how to ask it a useful question.
PDFs make the limitation concrete. Searching extracted text with BM25 is possible—the earlier benchmark does exactly that—but it offers different access from multimodal retrieval of the document's visual content. An agent looking for evidence in a table needs tools and queries suited to that evidence. Familiar text-search habits should not determine what the system can see.
A larger window still needs someone to choose what belongs in it
Context is finite and costs money. Clavié imagines a hundred-million-token window to emphasize that even a dramatic expansion would leave the problem of organizing legal knowledge across jurisdictions and specialties. The example is a scale argument, rather than a measured inventory of legal text. More capacity does not determine which facts matter or which dependencies the agent must investigate.
His proposed division of work gives those decisions explicit places. The main agent decomposes the task; searchers gather and organize relevant knowledge; memos bring the findings back for synthesis. Increasing context can supply more room, but this architecture decides what work happens before information reaches the answering agent. That is the design problem knowledge agents must solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The retrieval company and search tooling discussed by Clavié.
Further reading
Defines oracle retrieval and distinguishes human results with oracle evidence from BM25-constrained results. Its other configurations and scores differ from those reported in the recording.
- Our Research Vision, Part 1Reference
Explains Mixedbread's emphasis on useful retrieval research, serving costs, latency, and practical efficiency.
Read the complete timestamped transcript
- 0:13
Okay, so hi everyone. I'm gonna give the quickest introduction to myself. I'm Ben Clavié. I work at Mixedbread, where we do retrieval. I'm French, and I live in Tokyo. And today I'm gonna talk, talk to you about the fact that agents should do knowledge work, and so we should design them like knowledge workers. Like, we should design them like Knowledge Agents and not coding agents, and I'm going to explain the difference and why I think that's important. It's a bit of a hot tech talk, but let's start now. So the first thing is, like, first agents give us fun trivia, and I'm
- 0:43
talking, like, early agents, twenty-twenty to agents, back when all you had was, you know, RAG, but agents couldn't even do tool call back then. So all you had is, like, you had an if statement, you did search, you got, like, cool, you could talk to your PDF. That was the very first form of agentic work. It was not very useful. We're not gonna talk about that for long. What came next was programming agents, and that's been all the rage. Like, once agents started being able to search, actually properly search, actually properly understand things, carry tasks out, call tools,
- 1:13
we started designing coding agents, and coding agents are a big thing. I don't think there's anyone in this room that does not use coding agents. We all use CodeCard, Codex, et cetera. And that's a form of knowledge work. But agents were not knowledge workers at the time. Like, agents were coding agents. And now they're becoming knowledge worker. And by knowledge worker, I mean that knowledge work is a big superset. And kind of every workflow you've thought of before is a form of Knowledge Agents just because of the nature of knowledge. So it's like, if you have an agent
- 1:43
that's a lawyer that's looking for legal documents, if you go financial agent, if you're looking for, you know, medication information, like you've got a lot of people on Twitter that try to do those self-diagnosis, and you've got just a lot of medical usage. All of that is coding agents. We're getting an agent. It's trying to find-- Sorry, Knowledge Agent. It's trying to find knowledge, trying to make use of knowledge. And coding is part of that, of course, and even the small RAG bit that we talked about, like, in the first slide is part of that. But that's a very, very small proportion
- 2:12
of the actual full thing. Like, there's so much more to knowledge than any one domain. And what even is knowledge work? 'Cause I'm saying that it's important. They do knowledge work. Coding is knowledge work. And I think there's two ways to define it, in my opinion. One of them is knowledge work is work where your main input is information. Like, your main input is not an actual physical material. It's not something that you can touch. It's knowledge. It's information. And the nature of knowledge work is that you process this
- 2:42
information, which is by nature very ambiguous, very diffuse. And the main output you get from that is something actionable. It's a judgment. It's a decision. It's some-- If it's a lawyer, you're gonna get, you know, their findings on your case, and they might plead for you. You're going to get an actual actionable thinking item, like something still not tangible, but that exists. That's knowledge. There's also a tautological definition which makes sense here, that if you need search, it's a knowledge problem, and if it's a knowledge problem, you
- 3:12
need search. So it's very self-defined. And in the real world, that's basically most of the work that we see in the service economy is a form of knowledge work. Like lawyers, knowledge workers. Academics, knowledge workers. Actuaries, knowledge workers. Software engineers, researchers, also knowledge workers. And the fact that there's so much knowledge work in society has contributed to, like, an ever-improving structuring of knowledge work. Like, there's actually very, very well-defined workflows for how we should do knowledge work, for how knowledge works in
- 3:42
itself, and how we evolve that. But so far, agentics kind of focus on the special case, and that's try to generalize from it, and that special case is coding and software engineering. And the thing is, code is knowledge, but not all knowledge is code. And code's a very, very unique form of knowledge because it has very durable cues. Like on-- in a code base, there's gonna be a lot of references to, like, an identifier or a file or a path. And okay, when we vibe code, that can change, but most of the time it's not going to change all that
- 4:12
much. Like, all the things are, like, very, very durable. Then you've got that surface obviously is, like, now in grepable. Like, there's keywords, there's method definition, there's a lot of things that by definition you can grep in code. And the task-- And this one's actually very important, and we don't talk about it a lot, but people are like, "Oh, why is grep good enough for programming?" Or, "Why can an agent do programming?" And then you're telling me it can do, like, deep research for a legal question. And one of those is because we don't realize it, but, like, when we interact with coding
- 4:42
agents, we are giving them extremely narrow tasks. Like, we're not actually expecting that much from them. Like, everything's always kind of about a feature or about, like, a given ticket. Like, there's a task at hand. You're not going to tell the agent, like, "Discover a new programming paradigm and then implement it in this new app." I don't know what it's gonna do. Good luck. But in knowledge work, that's often the case. Like, first of all, you don't have those durable cues. Like, the meaning is always implicit. And more importantly, the same cue can mean a lot of different things. We don't
- 5:12
have function definitions in knowledge work. Like, if you see thirty days, if your agent's looking for thirty days, is it a deadline? Is it a grace period? Is it a retention rule? Is it even in the same domain? Like, are you searching for thirty days on a contract and you're getting medication? There's a lot of contextual information, yeah. But more importantly, the search starts from an intent. Like, even if you're doing, again, legal example, if you're asking about, like, a specific rule that you want to apply to a specific domain, you're going to need to look at the international norms that apply, and then do they apply in
- 5:42
this case? There's a lot of conditional information that is not predefined in the task. Like, that's all up for the agent to find. And so non-code knowledge is very contextual and meaning-driven, which is much harder than code. And that's led to the fact that none of what I'm saying is new. Like, people have been doing knowledge work for a very, very long time, and that's resulted in, like, two endless loops. So you have a tool loop, which is... At the start, we were, we were, like, talking. Then at some point, some guy was like, "We
- 6:12
should write stuff down." Then in Alexandria, we had the Pinakes, which was the curator of the Library of Alexandria, came up with an idea that maybe we should have a way to catalog all of the books we have. Then we developed writing, then we developed bibliographies, then we ended up with, like, the current version of the, like, Dewey system for libraries, and nowadays we have search engines. But we also had an organization loop, which is joint, but also disjoint from the tool one, which is it used to be the one gifted expert. Like,
- 6:42
we've all heard of the polymath of the past, the person who just knew everything about one domain or all domains, and you just went to them if you had information. But that doesn't scale, so we ended up with, like, monasteries, which were like guardians of knowledge, and then we had universities, and then we ended up creating the bureaucracies, and now we ended up creating the modern organization of work where we have very specialized firms. Like at hospitals, you've got the doctor, you've got the senior doctor, you've got the nurse practitioner, the nurses, the healthcare assistants, and all of them kind of like specialize on different levels of tasks, and that's
- 7:12
a really good form of optimization. But the thing is that it's actually just the one loop. Like, I'm showing two loops here, but they're actually just the one loop, which is we have new knowledge, and new knowledge means that we need better tools, and better tools mean that we end up creating new workflows, new roles, like we need people that are trained to use those tools, people that understand what the new tool does. If you have a guy that knows how to go to the library and you're like, "Okay, use Google," you need the knowledge of what Google is. Like, that person needs to be taught
- 7:42
that's a search engine. You can just type stuff in it. There's no need to physically go there. And that means you retrain. You get new knowledge workers who are more efficient, so they create more knowledge, so we need new tools, and so on and so on. So both the tool loop and the organizational loop are actually just this one self-optimizing loop that kind of like triggers the other endlessly.
- 8:01
And the thing about, like, tooling and optimization is that they're not neutral add-ons. Like, I said that we keep optimizing tools and things come up and we create new things out of those tools, but that's never actually a neutral thing. Like, tooling is not just, "Oh, my search is 5% better." The fact that we have a tool or the fact that we don't have a tool is what decides if a task-- not if the task is possible, because you can do things without the right tool, but if the task is actually scalable and can be carried out cheaply, because something being cheap means it can scale. And it's like, yes, of
- 8:31
course, if you go to the Library of Alexandria before the Pinakes, you can find your manuscript somewhere. Like, whatever you're looking for is there. It's probably going to take two or three weeks. So you're going to really, really, really need that knowledge. But if there's a library catalog, it's gonna take you ten minutes, and now, like, it's way easier to just, "Oh, okay, I need to know something more about this, so I'm gonna search for it." Likewise, if you have a map directory or if you even have a map in the first place, which in itself is a tool for information, then exploring the world is a much better idea. Like, you're not gonna rely on
- 9:02
randomly discovering America on your way to the Indies. You know, like, you know where you're going. And likewise, if you have, like, a multimodal search platform, then you can search millions of PDF in a way that we couldn't before. So now there's a lot of use cases where you are like, "Oh, it's in the archives. I'm not gonna touch that," that become actually useful.
- 9:20
And in practice, this kind of looks like that, and I'm getting into the more technical stuff here, which is on a simple deep research task. So this is the Brass Compass leaderboard, which is made to evaluate the quality of search tools on a very bounded deep research task. You have two hundred thousand documents and you have, like, specific queries. Yo talked about this this morning, and it's a really useful benchmark to, like, analyze queries. And what we see is that, like, okay, a bad tool. So that's the thing that people often rant about. You will see that there's two BM25 here. There's two,
- 9:50
like, optimized and unoptimized, and that's because quite often people will tell you BM25 is not great. And the reason they'll tell you BM25 is not great is because there's not one BM25. There's hundreds of them. It's a way to do lexical search. You should always optimize your baselines. You should always, like, optimize what you're getting. And so what you see here is, like, a badly optimized tool is useless. Like 60% accuracy. You're not going to trust someone that's right 60% of the time. You're just going to do it yourself. When you start optimizing the tools, you can see we go up to seventy,
- 10:20
eighty, and then the best, the actual best is Hybrid Harness. It gets to ninety. But that's maybe not the most interesting part because we start kind of plateauing at one point. Like the jump from 88 to 9.8 to 90.2 is in run variance. That doesn't matter. What matters here, however, is that 90.2% accuracy, you reach it with 20% fewer tool calls, and that's huge because in practice, that's 20% fewer tokens, 20% less fewer resources that you use. That's basically 20% free
- 10:50
cash. And if you compare it to the unoptimized baseline, you're spending, like, 5% of what you were spending in the first place. So the tool is actually what makes the task worth doing. Nobody would keep using that tool if it takes 25 calls. But if it takes 8 calls, we're like, "Oh yeah, cool. That's a workflow I can introduce."
- 11:09
And the second part, which goes with tooling and I think is just as important because Brass Compass in the previous slide is interesting, but it's easy. It's a hundred thousand documents. It's just text. It's just the one question. And it's not really that open-ended. It's just a bit convoluted. But when you're actually doing, like, real-life knowledge work, there is that workflow that you see that I tried to doodle, which is you have a client. They come to the big shot. They come to the lawyer. That's the partner of the agency. And they're like, "Okay, this is my situation. That's my problem." And they meet together. But
- 11:39
then what the partner does is they're not going to be the ones doing all the legal research. They're not going to be the ones, like, doing every single step of the problem. What they'll do is kind of understand that, like, okay, this person is, has this problem. That's gonna cause them that. Those are the facts. I need the relevant laws to this, that, and so on aspects. And then they've got paralegals, they've got assistants, and the assistants are going to be doing this research. They're going to be using, like, the same sort of tools they've been trained to use, and they're going to produce memo and notes, and then they're gonna give that back to the big shots that
- 12:09
maybe they research, like, one clarification point, but they mostly rely on what their searcher agents, if you want, like the assistants have found for them. And that's the response that you're gonna get. And that's echoing the point I made before, which is in code, when you're using code code, you're kind of doing that work yourself. You've already broken down the query, you know, like, you know what you want to do. You've got a linear ticket, you've got something that you're giving the agent. In the real world, you've got a client that's got a very open-ended problem, and you need to break it down yourself, and your agent needs to break it down itself, and then needs to use sub-agents
- 12:39
that do this research.
- 12:42
And this is how better tools and organization work together because this one is MatQA, which is another form of knowledge benchmark. MatQA is something that Hugging Face and Snowflake jointly released, and it's PDF-based enterprise task. It's got PDFs, and it's got OCI, OCR versions of the PDF. And the current state of the leaderboard really show that, like, both tools and organizations are necessary. And you can see that in the fact that with BM25, however optimized it gets, the
- 13:12
human and Gemini 3 reached the same ceiling. And that doesn't mean that Gemini 3 is as good as a human. That means that even a human cannot get the right information given unlimited searches with BM25. So you have the tool ceiling, and you need better tools to go forward. That's the, like, tool optimization part of the loop. Thankfully, we've got better tools. You know, we've got, we've got models that can handle PDFs. We've got Vision. We don't need to rely on OCR text. And what we see with that is that we get another jump, which is Gemini and the Mixedbread
- 13:42
search tool, which is fully multimodal, so it can read the PDF. You get the tables, you get all that nice stuff in your search, and that gets us a big jump in accuracy. But the interesting part about, like, that doesn't work. The-- Well, that doesn't work. That does work, but that doesn't work as well as we would like because why is my agent getting eighty-eight point nine if the human's getting ninety-nine point four? Like, that's ten percent I'm leaving on the table here. But that-- I don't understand. That's an agent. It gets, I think, ten turns in the benchmark, so it's a fully agentic system. It gets to think about its results, and yet it's missing
- 14:11
performance. And that's where we introduce the Mixedbread search agent, which is exactly that breaking down of work we saw earlier, where we basically tell the, the main agent, the one answering the question, being like, "Okay, that's, that's a big topic. There is, like, thousands of PDFs. You're not gonna, like, search yourself. Just please break down the problem for me. Please write queries about the aspects that you think are important to answer the actual query." And we get searcher that go off on their own, and they find the right results, and they bring you, like, a little memo to your agent, and then your agent actually answers that. And
- 14:41
that gets the accuracy up by three point five points. And that doesn't sound like a lot, but I like to think of it as, like, an oracle gap. And the oracle gap is the difference between perfect documents and your search system. And the oracle gap here is about, about ten points before using the agents, and it goes down to six points after using the agents. So that means that we have about a forty percent reduction in mistakes. Like, the gap between humans and agents goes down by forty percent just by, like, having a better architecture to search for it.
- 15:13
And I think this is the end of my slide 'cause I'm running out of time, and that's perfect 'cause that's my takeaway slide. And what I want you to, like, get from this talk is that we know how to design better knowledge work for humans, and AI agents really benefit from this pattern. Like, we have designed this. We know how to do this. Humans have worked on this for centuries. Like, people have always needed more knowledge. The empires used to have librarians. We have paralegal. We've got legal firms. We know exactly how legal, the legal industry has figured out paralegal. The medical industry has
- 15:43
figured it out. And none of it looks like programming. Programming is a very different system because it's a very specific use case, and we should really learn from the knowledge world to know how to design agents that will do work for the knowledge world. And then you must not, like, overfit on tools because tools don't exist as a way to do things by themselves. Tools exist as a way to overcome ceilings. You want a better tool when you see that you're hitting a ceiling, that your performance is not where you want it to be. So we design better tools to overcome the
- 16:12
ceilings. And more importantly, the tools need to be co-designed with the agents. Like, the agents need to know how to use tools because one thing you would often see is agents will try to write grep queries because grep's everywhere in the training data. BM25, they will end the query in the data, and that's not always what you need. Sometimes you need semantic search of a PDF, and you can't grep a PDF. You can't BM25 a PDF. You need to, like, write a better query. So it's very important that your agentic harnesses or even your agentic models know that they have got more than one tools, and it's
- 16:42
about primitives. Grep's a primitive. BM25 is a primitive. And semantic search is a primitive. And all of those need to be, like, very well-trained. Like, the models need to know about all of them. And the last one is that the right orchestration of search will get you much better results because context is a finite resource. And even if we get to a model that's got, like, a hundred-million token context, A, that's gonna cost you a lot of money, and B, that's still nothing. You're not even getting half of, like, one state's legal
- 17:12
code, let alone the US, let alone international law, let alone specialized course, et cetera. So you need to have a way to break down your task, and you need to have your orchestrator or your main agents and people that can actually organize the knowledge for them.
- 17:26
And yeah, cool. So we've got two minutes for questions.
- 17:34
Thank you.