The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
AI Engineer World's Fair 2026 · 18:28
← All organizations
Retrieval infrastructure for AI agents
Hornet.dev
Hornet.dev is building Hornet, a retrieval engine for AI agents. Its premise is that agents generate a different search workload from humans: they issue long, structured queries, search repeatedly inside reasoning loops, and read documents to decide what to investigate next. Hornet is being designed for workloads ranging from a single agent searching a scoped collection to multi-agent systems issuing parallel queries over web-scale datasets. Its schema-first APIs aim to make requests predictable and reduce errors and wasted tokens. The company describes support for large and small models, with deployment planned as a managed service, on-premises, or in a customer's own cloud, beside the agents and their data.
Jo Kristian Bergum, Hornet's CEO and co-founder, develops the company's technical case in This is what agentic retrieval looks like. His analysis of published GPT-5 + BM25 trajectories from BrowseComp-Plus covers 830 questions and 19,279 search calls, with a median of 24 calls per question. The median query contains 10 terms, compared with two in the historical AOL human-search dataset; phrase quotes appear in roughly 98% of agent sessions. These findings describe a particular benchmark workload, but illustrate why short, single-query search assumptions can be a poor fit for agents.
The analysis explains two mechanisms behind that mismatch. Each search result becomes context for the next step, so a useful document missed early can change the evidence available throughout the remaining reasoning session. Agents also write operators such as site: and phrase quotes into their queries; if the backend ignores that syntax, an agent may reason as though a constraint was applied when it was not. Hornet's proposed retrieval infrastructure responds to these iterative workloads and explicit query requirements. Its website still describes the product as under development and invites prospective users to receive updates and help shape it.
Explore the recordings
The supplied Hornet.dev archive contains one recording: Jo Kristian Bergum’s talk on BM25 for agentic search. Use the paths below to explore its retrieval design, evaluation lessons and performance claims. This guide describes the recording rather than company history; it does not establish current product capabilities or performance.
Start with the agent search loop
Watch The unreasonable effectiveness of BM25 for agentic search for Bergum’s account of how a capable model, a retrieval harness and a search engine work together. He argues that models’ general knowledge and iterative query formulation make lexical search more useful. Follow this path if you want to understand why retrieval choices should be considered within an agent’s workflow.
Explore evidence acquisition and workspace design
Return to the talk for its BrowseComp+ evidence-placement experiments and discussion of file-system workspaces, progressive disclosure and tools such as grep. Bergum interprets the experiments as showing an evidence-acquisition bottleneck in that setting. These parts connect corpus retrieval to how an agent selects, inspects and refines evidence.
Examine evaluation and performance claims
Use the recording to examine Bergum’s warning that BM25 parameter choices for long documents can distort retrieval comparisons, and his case for evaluating completed tasks alongside single-query ranking metrics. He also presents claimed Hornet throughput and latency advantages over anonymized engines on 100 million web documents. Treat those as recorded benchmark claims, not independently verified comparisons or facts about today’s product.
1 talk
Newest first1 speaker at AIE
Affiliations reflect their AIE appearances, not necessarily current employment.
