AI Engineer World's Fair 2026
The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
Read the talk
Why BM25 works so well for agentic search
Jo Kristian Bergum explains how knowledgeable models, repeated queries and file-system workspaces give an old lexical scoring function a new role—and why the complete search loop matters more than one ranked list.
From a talk by Jo Kristian Bergum
At a glance
Ideas worth remembering
Agentic retrieval depends on the model, harness and engine together. High accuracy with evidence supplied directly does not guarantee that the model can find it through search.
A BM25 comparison needs its configuration and workload. Bergum attributes substantial BrowseComp+ accuracy differences to parameters poorly suited to long documents.
Literal matches help agents find specific identifiers and understand returned results well enough to refine their next query.
A file-system workspace supports progressive disclosure: show titles and snippets, then let the agent inspect selected documents with familiar tools.
Evaluate the complete trajectory by task success, while measuring engine latency, throughput and cost separately.
Search inside an agent loop
An agent writing code or doing deep research eventually needs information it does not have. It must find that material before it can finish the task. Jo Kristian Bergum, CEO of Hornet.dev, calls this agentic search: search inside an agent loop. After more than twenty years working on retrieval, he sees a new role for BM25, a roughly thirty-year-old lexical scoring function.
The system needs three parts, each responsible for a different piece of the work:
- Model: Formulates queries and uses tools to obtain missing information.
- Harness: Exposes retrieval through an interface the model can operate, such as tool calls or code mode.
- Retrieval engine: Executes searches efficiently, potentially across billions of documents.
This separation matters because model capability alone does not supply a search system. The model needs an interface for expressing its information need, and that interface needs an engine capable of searching the underlying collection. Agentic retrieval depends on how those parts work together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An unchanged scoring function gets a more capable user
BM25 stands for Best Match 25. Bergum recounts a series of scoring experiments in which the twenty-fifth worked best, leaving its number in the name. The function scores a document against a query through interactions between their terms. That score is a proxy for relevance, allowing the system to select the top K documents.
Scoring and execution solve different problems
A straightforward implementation would score every document, then keep the highest-scoring results. At large scale, the work lies in finding those results efficiently. Bergum points to decades of algorithms for accelerating top-K retrieval. BM25 names the scoring function; it does not identify the algorithm or engine used to execute it. Engines using the same scoring rule can therefore have different serving costs and performance.
Bergum’s explanation for BM25’s comeback begins with the user. A language model already knows entities, companies and dates. It can draw on that knowledge to supply specific search terms. The scoring function has stayed the same, while the client has acquired a much stronger ability to describe what it needs.
BM25 traditionally served as the baseline against which researchers compared more elaborate neural methods. Search evaluation likewise often assumed a person issuing one query and scanning ten blue links. An agent can issue many queries. That makes the single search result a less complete description of the system’s usefulness: the agent can read a response and search again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
BrowseComp+ separates finding evidence from using it
Bergum turns to BrowseComp+, a deep research benchmark with 830 long, riddle-like questions—closer to a pub quiz than a short keyword lookup. The corpus contains roughly one hundred thousand web documents. A model receives a simple Search tool: it supplies a query string and receives snippets. Each question has a reference answer, allowing the benchmark to check whether the complete loop produces the expected answer.
The context window is a small working space
Bergum compares context windows to floppy disks. A disk held about 1.4 megabytes; he estimates that models can use around 350,000 tokens before quality starts to degrade. The token figure is his working judgment, rather than a universal threshold. The analogy explains why retrieval remains necessary: the system must choose useful information for a bounded working space.
BrowseComp+ makes that selection problem visible through two conditions:
- Evidence supplied directly: Put the documents needed to answer the question into the model’s context. Accuracy is high, including with older models.
- Evidence found through search: Give the model a retrieval tool and require it to obtain the evidence. Accuracy falls, because success now depends on query formulation, the harness and the retriever.
Bergum concludes that reasoning is not the main bottleneck in this experiment. The model can use the evidence once it receives it; finding that evidence introduces much of the difficulty. The comparison does not isolate the retriever as the sole cause of failure, since the tool interface and the model’s search decisions also change what reaches context.
A riddle consequently becomes a search trajectory. The model executes a query, reads the response, reformulates and repeats until it finds the answer or fills the context window. Bergum expects evidence selection to remain important as models improve: a more capable reasoner still needs the relevant documents in its working space.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agent queries change the workload—and the baseline
Hornet examined these search trajectories and compared model-written queries with human queries in the AOL search logs. Bergum describes human queries as short, a pattern he also recognizes in more recent logs. The model writes longer queries and reaches for phrases and operators such as site:. Hornet’s analysis of the agent query workload develops the comparison. These are different inputs from the few terms a person typically types into a search box.
Which BM25 configuration is being compared?
Bergum notes that BM25 has two hyperparameters, then challenges the baseline used in the original BrowseComp+ research. In his account, later research found that those parameters were poorly suited to the benchmark’s long documents. Embedding-based methods looked substantially better against that baseline, while changing the BM25 parameters had a dramatic effect on overall answer accuracy.
The qualification is specific: a poorly configured baseline can distort the comparison between retrieval methods. Bergum does not give the parameter settings or numerical accuracy changes in this discussion. His argument supports examining the configuration and document characteristics before concluding that lexical retrieval is weak; it does not establish that every tuned BM25 implementation outperforms embeddings.
Why literal matches help the agent
Bergum gives three reasons for BM25’s usefulness in this workload:
- Specific identifiers: Names, entities, ZIP codes and SKUs can become precise query terms. He argues that embedding-based retrieval has more difficulty preserving these distinctions when the literal identifier matters.
- Cost and availability: BM25 avoids the embedding-inference step and has mature tooling. Bergum contrasts that with encoding text using embedding models that can have about eight billion parameters, along with the infrastructure needed to serve them.
- Inspectable feedback: The model can see the literal terms and phrases in returned text and understand why a result appeared. That gives it information for reformulating the next query.
Inspectability matters because results become inputs to later searches. If the agent recognizes which words led to a document, it has a concrete basis for changing its query. Literal matching therefore contributes both to finding a candidate and to deciding what to try next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve across the corpus, then inspect files selectively
Bergum next introduces Scaling Direct Corpus Interaction via Dynamic Workspace Expansion, work from Jimmy Lin’s group at Waterloo. The motivating problem is web search for agents over potentially billions of documents. Those documents cannot all enter context. A retriever selects material from the large corpus and places the retrieved documents in a workspace, which Bergum likens to a search results page for agents.
Titles and snippets lead to deeper reads
Organizing the workspace as a file system enables progressive disclosure. The model first sees a document’s title and a small snippet. It can then decide whether to read more. Full documents remain available for inspection without all being inserted into context at once. The workspace separates having retrieved a document from having read its contents.
The model can inspect that material with familiar coding-agent tools: grep, ripgrep, sed and awk. The retrieval engine searches the large collection; file tools help the agent examine the selected material and manage what it reads. The resulting architecture combines retrieval infrastructure with a sandbox or virtual file system and a Bash interface.
The useful data flow is corpus → retrieved files → titles and snippets → selected deeper reads. BM25 supplies corpus-scale retrieval, while file tools give the model a way to inspect the results. In Bergum’s closing argument, BM25 and grep work well together because both expose literal text matches the agent can understand.
Build around capabilities models already practice
Bergum describes this as a pragmatic adaptation to model training. Developers of frontier models invest in coding, Bash and tool use, so a retrieval interface built around those capabilities may benefit from that investment. He expects this direction to improve with models, while leaving open whether more capable systems might eventually work directly through a browser. The design is a bet on a useful interface, not a guarantee about every future model.
Potentially billions of documents, too large for context.
Retrieved documents stay in a file-system workspace. The agent sees compact previews and chooses what to inspect more deeply.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Judge the search trajectory and measure serving efficiency
Traditional retrieval evaluation often begins with one query, one ranked list and a metric such as nDCG. Bergum argues that this gives an incomplete picture of an agent that can reformulate queries, expand its search and use later results. His proposed endpoint is the task itself: for question answering, does the agent get the answer right?
His dismissal of classical evaluation is forceful, but the mechanism behind it is concrete. One ranked list measures one retrieval step. The agent’s answer emerges from a sequence in which each response can change the next query. Task success captures the outcome of that interaction between model, harness and engine.
The engine still has to serve the workload economically
Hornet is betting on BM25 as a fundamental retrieval primitive and on executing it efficiently. Bergum presents a comparison with anonymized engines using the same type of hardware and one hundred million web documents on a single node. He claims Hornet achieves more throughput for the same spending, with potential savings for companies building search infrastructure.
The performance claim remains qualitative here. Bergum corrects his explanation of the vertical axis from queries per second to latency; his explanation does not establish the horizontal axis, numerical values or competing engine identities. Those omissions prevent a quantified speedup or reproducible cost comparison. Serving efficiency also answers a different question from answer accuracy: returning results faster does not establish that the agent found the evidence it needed.
Bergum ends with the combination he wants builders to remember. A model can read and write quickly, draw on general knowledge and reformulate queries, making simple tools more powerful. BM25’s parameters, implementation and performance still need attention. Corpus-scale lexical retrieval followed by grep-style inspection gives that capable user understandable results to work with—and another search to make when the first one is incomplete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Bergum’s retrieval infrastructure project, built around agent search workloads.
Hornet’s analysis of BrowseComp+ search trajectories, query lengths and operator use—the workload comparison discussed in the recording.
Further reading
A separate controlled comparison of retrieval harnesses. It measures gold-document recall and token cost, rather than end-to-end answer accuracy.
Read the complete timestamped transcript
- 0:12
So great being here. Uh, I'm Jo Kristian Bergum. I'm the CEO of, uh, Hornet.dev, and I'm here today to talk about the unreasonable effectiveness of BM25 for agentic search. So how many of you heard about BM25 before? Is it new, or is it... Oh, quite a few, so that's great. Uh, I'm also watching the World Cup. Norway is playing their, uh, against Ivory Coast second half. Norway is leading, so that's good. So yeah. And at Hornet, we are building
- 0:42
retrieval infrastructure for agents, and, uh, I've been working on search and retrieval problems for a long time. As you can tell, I'm gray-haired. Uh, been working in this space for more than twenty years. Um, and in the talk today, I'll talk about why this kind of thirty-year-old lexical scoring function is, uh, making a strong comeback. First, I will talk a little bit about what I mean by agentic search or agentic retrieval and kind of
- 1:12
define that for, for you, all of you. Um, my definition is that agentic search is essentially search inside a agent loop. So you have an agent that is trying to accomplish a task, write a coding or, uh, do some deep research or whatever task. And inside that, you have some kind of information need for the agent, right, in order to get that task done successfully. And you essentially need three things, uh, to kind of build a good agentic
- 1:42
search system, and that is you need a capable model, a model that is able to use tools and is able to formulate queries. You also need a harness around the model in how you kind of expose the retrieval and the search functions to the model. There's different ways to do that. It could be through tool calling, or it could be through code mode. Um, Edo here, uh, demonstrated what I call code mode for exposing retrieval
- 2:12
infrastructure. So that's the harness part. And then you also need, uh, a retrieval engine to be able to do searches efficiently, potentially over billion-scaled, uh, document sets.
- 2:27
And to define BM25. So BM25 actually stands for Best Match 25. So there were some researchers doing plenty of different experiments, and experiment number twenty-five turned out to be the best one. So that's, that's the background of the name. It's essentially a scoring function. So you can imagine you have a query, and you have a document, and you calculate some kind of score by interaction between the query terms and the document terms, and you come up with a score, and you hope that this score kind of is
- 2:57
a good proxy for the relevancy of the document with regards to the query. Right. So one way to calculate BM25 would be to take all the documents and score each of them, you know, and then figure out what are the top K documents. And then there's, like, thirty or forty years of interest in how you accelerate that type of retrieval. Top K, lots of different algorithms. We also invest a lot in that. I'll show something in this direction, but BM25 is the scoring function, and there is a
- 3:27
way to kind of accelerate top-K retrieval. Uh, BM25 hasn't changed. It's the same scoring function. But the change here is really that we got a more powerful user. Uh, Edo talked about general knowledge. The LLMs today have a lot of general knowledge. They know about entities. They know about companies. They know about dates. They know a lot, right? And by using that kind of implicit knowledge that is built into the parametric model,
- 3:58
uh, they essentially become very good at search. Um, and that's the really change, uh, here now that we're-- is kind of making BM25 more relevant. And BM25 used to be a kind of a baseline function. Any information retrieval research would include a BM25 baseline, and then you would put something fancy, advanced, neural, fancy stuff, and then you would compare it with BM25. I also think it's interesting in how we evaluate search
- 4:27
before, because you will simply look at ten blue links, and you, like how scan it, and you compute some metrics. A lot of that is now going away because the agent is not really... It's kind of powerful in the way that it can type out a lot more queries than a human can do, so it's less relevant to think about, uh, evaluating these systems just by a single-shot query.
- 4:53
And this is one of my kind of favorite benchmarks out there. I like to talk about benchmarks. So BrowseComp+ is a deep research benchmark, uh, published in a paper last year. And it has almost or exactly eight hundred and twen- thirty questions. These are riddle-like. Think about it like a pub quiz. Do you have pub quizzes in the US? Yeah. Okay, great. So like kind of a riddle type of questions, quite long.
- 5:24
And the agent, the harness of this kind of-- or the protocol of this benchmark is that you have a model, and it gets a very simple tool called Search, and it accepts a query string, and you return some snippets back to the model. And the corpus is about hundred and five thousand or a hundred thousand documents, so that's kind of tiny, and these are web documents. And the end-to-end accuracy, um, all of these questions have a golden reference answer, and you can
- 5:54
kind of check if the model and the entire loop produces that exact answer.
- 6:01
Um, but why kind of, why do we need retrieval? I like to compare context windows with floppy disks because I'm old. In the '80s, right, we installed these kind of games on our computers using floppy disks. So one kind of... You guys are so young, so you don't probably have this kind of nostalgia, but one floppy disk could fit about one point four megabytes of data. And the current models, before they start degrading in quality, that's I-
- 6:31
in my opinion, around three hundred and fifty thousand tokens. So that's one floppy disk of data, right? Um, so you need retrieval in order to, um, to fetch the information that you actually need to put into the context window. And BrowseComp+ really demonstrate how, uh, retrieval quality affects the end-to-end accuracy of the task, right? The end-to-end accuracy here is essentially, is the model equipped with this
- 7:01
search tool able to answer the question, right, the riddle-like question? And if you artificially just stuff the evidence documents that is needed to answer this question into the context window of the model, the accuracy is really high, right? So reasoning is not the bottleneck. The model, given the evid-evidence up front, answers the question with a very high accuracy rate, even GPT-4.
- 7:33
Um, but if you expose the model with the harness, with a retrieval tool, uh, that accuracy falls because it now depends on the harness, it depends on the model's ability to formulate queries, and the retrieval quality of the retriever. Um, so for me, this is also quite important because even if we get perfect models, like models that are kind of AGI, you don't have to append, "Make no mistakes," you still will be limited to a context window that
- 8:03
is approximately a floppy disk, right? So you have to decide what goes into that context window, and I think retrieval is still very relevant as the previous slide showed.
- 8:14
And in this BrowseComp+ data set, one of these riddle-like questions become a search trajectory because the model execute the query, gets some response back, reads it, reformulates the query, and continues until it has kind of filled up the context window or found the answer, whatever comes first.
- 8:37
And we spent some time to investigate these trajectories, uh, to see how GPT-5 is formulating queries, and we found a lot of interesting aspects with that. Uh, we described it in a recent blog post as well. You can find it on Hornet.dev. And we like to compare it with, uh, AOL query logs. So AOL, AOL was like a service back in the day, had some
- 9:07
search interface, and they accidentally published, um, a very large sample of what people were searching for on the web, and they were quite short. And I have seen more recent query logs as well, and the user-human pattern are still searching with just a few terms. GPT-5, on the other hand, it's a much more powerful user. It has the general knowledge, and it can like, bam, write out very long queries. It can use
- 9:37
a lot of syntax operators that are kind of useful from... It has learned from web search or site operator phrases, et cetera. And this is a new type of workload.
- 9:52
And on BM25. BM25 has essentially two hyperparameters that controls various aspects of the scoring function. And I talked about having a baseline, and BM25 was usually a baseline, and BrowseComp+ as well has a baseline with BM25, but it turns out that that baseline is terrible. So when you look at fancier techniques, embedding models, what have you, um, it stands out
- 10:22
as a much better retrieval paradigm than BM25 if you look at the or-original paper. But more recent research shows that the parameters that were used in the BrowseComp+ research paper was not really adequate to handle these kind of long documents. So I like this, "Which BM25 do you mean?" Uh, because it has a quite dramatic impact on that specific benchmarks on the overall accuracy.
- 10:50
And why is BM25 now more powerful with the new user? So I mentioned the general knowledge of the user and that he can type faster and more, be more specific as a more powerful user. And exact matching is still relevant, right? Because the model knows names, entities, uh, zip codes, uh, SKUs, what have you, that is not so easy to represent with an embedding model which kind of encodes all the tokens into a fixed vocabulary.
- 11:21
Uh, it's also relatively cheap, especially if you take into consideration the cost of doing embedding inference, right? Some of these embedding models have like eight billion parameters, and you encode text, and you have to stand up infrastructure for this and what have you. So it's quite simple, and also the tooling, uh, in the overall ecosystem is quite good. Um, so it's, it's readily available. And it's also very easy for the model to inspect the results and understand why a certain query
- 11:51
formulation returned the result it did, right? Because you're matching literal terms and phrases and things like that, which can hi- help it kind of reformulate the query. So these are the three key things. And now into more hot topics on, like, what is all you need. Um, and this is a very recent research that came out from Waterloo, from Jimmy Lin's group up there. They are doing a great job at the information retrieval research and
- 12:21
also on agentic search. They have a recent paper that I love. Uh, it's called Scaling Direct Corpus Interaction via Dynamic Workspace Expansion. So I'll spend some time on expanding this. So imagine you want to stand up web search infrastructure for agents. A lot of companies are doing that at the moment. We are also working with some of these companies to help them build infrastructure for powering that kind of use case. And there you have potentially billions of documents,
- 12:52
right? So that doesn't fit into the context window, so you obviously need retrieval, and BM25 is a good baseline. So you can retrieve information over that, and the results of this is, um, you can imagine this as a search engine result per page, SERP, for agents because you can place these documents that are retrieved from the retriever into a workspace. And if you
- 13:22
organize this workspace as a file system, you can, uh, play into the same things that you have around skills. You can have progressive disclosure because you can have the document, like the title of the document and a small snippet of the document, and expose that to the model, and the model can then also decide, "Oh, I need to read more of the document." And when it's doing that, it can use all the primitive tools that it's really good at using. You all use coding agents, so you all seen grep, and ripgrep, and sed,
- 13:52
and awk, and whatever it's doing to kind of manage context. So here you get the benefits of both worlds, and also you get to combine sandbox infrastructure, retrieval infrastructure, so, and VFS, and just Bash, and whatever. So it's also quite exciting, right? Because it combines all of these new type of paradigms that is happening at the moment. So I'm really excited about this type of direction. And it's also kind of a hack, right? To optimize for what the models are good at at the moment, right? Because all the frontier LLM companies are
- 14:22
optimizing their model for co- for coding, for Bash, for tool use. So if you put your kind of end-to-end task on the trajectory of that, you kind of, whatever, there's a new model, you know there's gonna be better at this as well, right? Right. So might be that when we get AGI, they can just use the browser. We'll see. But currently, this is a very powerful way to, to build, uh, retrieval infrastructure and a whole agentic search experience.
- 14:53
And when it comes to evaluation, right? I talked about this earlier as well. In traditional information retrieval, we were used to having just one query, one ranked list, compute nDCG, and compare it. That is no longer very relevant when the new user is a agent, because that agent can reformulate queries and do more queries and do expansion and all, all kinds of different stuff, right? So a lot of the classical information retrieval evaluation is now kind of dead.
- 15:23
Instead, look at, like, see if the model can perform the task it's, it's set to. And for example, for question answering, does it get the answer right?
- 15:35
And we at Hornet, we are betting on BM25 as one of the primitives, and we have set out on a vision to kind of have the best, most efficient way to evaluate, uh, BM25, because I think it's such a strong, fundamental primitive. And this illustration compares some anonymized engines comparing with Hornet on the same type of, um, same type of hardware with, um, web documents, hundred
- 16:05
million web documents on a single node. And as you can see, Hornet has a much more efficient implementation than the other engines and can do more throughput for the same type of bucks, which for a lot of companies that are building infrastructure at the moment for web search, et cetera, means a lot in savings. What is the Y-axis? The Y-axis is QPS. Sorry.
- 16:33
What's the X-axis? The Y-axis is... Oh, sorry. The Y-axis is latency. I'm sorry.
- 16:40
So four claims to take away from this talk. There's a new user. It's more powerful. It's able to type faster. It's able to read faster. It's able to reformulate queries, and it has a lot of general knowledge, which makes simple tools like grep and BM25 more powerful.
- 17:03
That's number one. Number two, which BM25 do you mean? Uh, there are differences in implementation, in performance, in parameters, so think about that. And also, why is it effective for agentic search? It's simply because it's explainable for the model, so the model can see and also use it in combination with grep, right? Because you have literal matches, and grep is almost about literal matches as well. And the combination of these two is a very
- 17:33
strong agentic search or agentic retrieval paradigm.
- 17:39
There's a lot of references. I think I will publish a talk, uh, or the talk will be published and also the, the slides. And if you hated it, you can, uh, tweet at me.
- 17:55
Um, there is not, I was told that there's not a room for, for questions, but happy to chat about retrieval. You will find me around the conference. Uh, probably best way to reach me is through my X account, and that's it.