---
title: "A 40-year-old search algorithm beats modern embeddings at agentic research. Only one version of it does."
description: "A Hornet CEO says BM25 is unreasonably effective for agentic search. The paper he cites disagrees, then a newer paper agrees. The condition between them is the real finding."
publishedAt: 2026-09-17
updatedAt: 2026-09-17
author: Ena Pragma
url: https://enapragma.co/field-notes/bm25-for-agentic-search-which-bm25-do-you-mean
tags: ["ai-operations", "agent-architecture", "ai-verification"]
---

At AI Engineer this month, Jo Kristian Bergum, CEO of the search company Hornet, gave a talk titled "The unreasonable effectiveness of BM25 for agentic search." The argument: a lexical ranking function from the 1980s, one that literally matches words, is a strong enough retriever for a modern deep-research agent, and the field's rush toward dense embedding retrieval undersells it. We checked the talk against ten external sources, both supportive and competing, including the two papers it leans on hardest. The finding is narrower than the title and more useful: a default BM25 loses badly to a tuned embedding retriever inside the very benchmark the talk cites, and a properly tuned BM25 paired with a strong enough model beats released dense-retrieval agents outright. Both are true. The talk's own closing line, "which BM25 do you mean," is not a rhetorical flourish. It is the literal, sourced answer.

## What the talk gets right, and what it gets slightly wrong

Start with the small correction, because it is the kind of thing that erodes trust if left in. The presenter calls it "Best Match 25" and says the algorithm is about thirty years old. Wikipedia's Okapi BM25 entry, the closest thing to a primary source on the name, says BM abbreviates "best matching," and attributes the underlying probabilistic framework to Stephen Robertson, Karen Sparck Jones and colleagues in the 1970s and 1980s, with the Okapi system implementing it at City University London through the 1980s and 1990s. That makes the function forty to fifty years old, not thirty. Neither detail changes the argument. Both are worth stating correctly.

The load-bearing numbers hold up. The benchmark the talk is built on, BrowseComp-Plus, is a real August 2025 paper with a corpus of exactly 100,195 web pages and 830 human-verified question-answer pairs. The presenter's hedged "about 105,000 or 100,000" and "almost or exactly 830 questions" match. Jimmy Lin's group at Waterloo, which he names as active in this space, is on the author list.

The central mechanism also holds, and it is the most important thing in the talk. Hornet's own March 2026 post on the benchmark reports that when every labeled evidence document is stuffed into a model's context up front (the "oracle" setting), GPT-4.1 answers 93.49% of questions correctly. When the same model has to find that evidence itself using an untuned BM25 baseline, accuracy collapses to 14.58%. Same model, same questions, same corpus. The only thing that changed was whether the model had to retrieve. That is the talk's claim that "reasoning is not the bottleneck," and the number behind it is a first-party read of a public benchmark, not a vendor's estimate.

## The paper the talk cites also argues against it

Here is the finding Scout's research turned up that the talk does not mention, and it comes from inside the paper the talk relies on. BrowseComp-Plus's own Table 7 compares end-to-end search agents on the original corpus with a matched model, SearchR1-32B. BM25 scores 3.86%. A tuned embedding retriever, Qwen3-Embed-8B, scores 10.36% on the same corpus with the same model. That is roughly 2.7 times higher. This is not a critic's cherry-pick. It is a controlled ablation, published by the same authors, in the same paper, that the talk presents as supporting evidence.

It would be easy to stop there and call the talk oversold. That would be the wrong read, for two reasons. First, a naive lexical baseline losing to a tuned dense retriever is the expected, unremarkable result the field already treats as its starting point. Nobody in information retrieval finds it surprising. Second, a more recent paper from an overlapping group answers the exact question the tension raises.

## The newer paper that resolves it

"Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?" was submitted in May 2026 by Tz-Huan Hsu, Jheng-Hong Yang and Jimmy Lin, the same Jimmy Lin who is an author on BrowseComp-Plus. Its abstract answers its own title: "a well-configured lexical retriever with sufficient retrieval depth can support effective deep research when paired with more capable LLMs." Pi-Serini with GPT-5.5 reaches 83.1% answer accuracy and 94.7% surfaced evidence recall on BrowseComp-Plus, which the paper describes as "outperforming released search agents that use dense retrievers."

The ablations inside that paper are where the actual lesson lives. Tuning BM25 alone, without touching the model, improves answer accuracy by 18.0% and surfaced evidence recall by 11.1% over the default configuration. Increasing retrieval depth improves surfaced evidence recall by a further 25.3% over a shallow setting. The code is public. So the same research lineage produced both facts: default BM25 is a weak baseline that embeddings beat handily, and tuned BM25 with a strong model is a top-tier agentic retriever. The variable that flips the result is not the algorithm's name. It is configuration and the capability of the model driving it.

That is also why the talk's closing question is the right one. "Which BM25 do you mean" turns out to be the difference between 3.86% and 83.1% on the same benchmark.

## Why a simple retriever works better than it should

The mechanism, sourced from Hornet's analysis of a public trace file rather than from the talk, is that a capable model does not issue one query. It issues dozens. Hornet's May 2026 post analyzed the published GPT-5 plus BM25 run on BrowseComp-Plus, which is public on Hugging Face and independently inspectable: 830 questions, 19,279 search calls, a median of 24 searches per question, a 90th percentile of 35, a maximum of 63. The model reformulates, narrows, quotes phrases, and tries again, inside a single agent loop.

One number from that analysis needs both of its halves to be honest. Hornet's headline is that GPT-5 uses phrase quotes in 98% of sessions. The same post's finer-grained figure is that 65.56% of individual queries are phrase-quoted. Both are true. The first measures sessions containing at least one quoted query; the second measures queries. Cite the 98% alone and a reader concludes GPT-5 quotes nearly every search, when the per-query rate is roughly two in three. The post also notes that GPT-5's median query length is past the 99th percentile of human queries in the classic AOL query log. The model searches like a power user, and a power user can get a great deal out of a literal-match engine.

Two claims in the talk are the speaker's own and should be held that way. The figure of roughly 350,000 tokens as the point where long-context reasoning degrades is stated in the talk as his opinion, and we found no independently published study naming that threshold. The chart comparing Hornet's latency against anonymized competitors on a hundred million documents is a vendor's own CEO presenting the vendor's own numbers against unnamed engines, and we could not find it published in written form. Hornet's three-part series on building search at that scale is real, so the engineering is not in question. The specific comparison is, until someone outside Hornet reproduces it.

## Where hybrid retrieval fits, and where it does not

The obvious response to "lexical or dense" is "both," and that is a live research direction, not just an industry talking point. A March 2026 paper proposes a reference architecture for agentic hybrid retrieval that combines BM25 with dense embeddings through reciprocal rank fusion, orchestrated by an LLM agent, on the grounds that "typical lexical or dense retrieval alone falls short." Its scope is ad hoc dataset search over sparse, heterogeneous metadata, not general web-scale agentic research the way BrowseComp-Plus and Pi-Serini are. It is evidence that hybrid retrieval is being seriously engineered in a neighboring problem, and it should not be cited as a data point on the specific question this talk raises. The two findings sit beside each other. Neither makes the other obsolete.

The honest state of the field, as of September 2026, is not a choice between "is BM25 dead" and "is BM25 unreasonably effective." Both framings are strawmen. It is that default lexical retrieval alone underperforms dense retrieval by a wide margin, a tuned lexical retriever paired with a strong model can beat released dense-retrieval agents, and serious groups, including one lab that appears on multiple sides of this exact question across different papers, are actively contesting the conditions under which each wins rather than picking a permanent side.

## What an operator does with this

The actionable finding has nothing to do with BM25 specifically. It is the 93% to 15% gap. Give a mid-tier model the right evidence and it answers over nine questions in ten. Make it find that evidence with a weak retriever and it answers fewer than two in ten. Money and engineering hours spent on retrieval quality currently buy more accuracy than the same money spent on a bigger model, and the Pi-Serini ablations put numbers on how much: tuning alone improved accuracy by 18.0%, and retrieval depth improved recall by a further 25.3%. If you are building or buying an agent that searches anything, the retriever's configuration is the first place to look when accuracy disappoints, before the model is.

The wider lesson generalizes past search. A capable model inside a tool loop can compensate for a simple primitive's limitations by using it dozens of times with increasing precision, which is why a forty-year-old word-matching function is competitive with a 2026 embedding model when the thing driving it is GPT-5.5. As models get better at using tools in a loop, the cheapest and most explainable primitive around them is worth re-testing under real conditions before assuming it needs replacing with something more sophisticated. The talk's title says the effectiveness is unreasonable. The papers say it is conditional, and they name the conditions. That is a better result than the title promised, because you can act on it.

Related field notes: [Chunking is not dead. The talk that says so smooths one of its own numbers.](/field-notes/chunking-is-not-dead-the-talk-that-says-so-smooths-one-of-its-own-numbers) and [Knowledge agents need a different shape than coding agents.](/field-notes/knowledge-agents-need-a-different-shape-than-coding-agents), the two other retrieval talks from the same conference we checked the same way.

## Sources

- [AI Engineer (YouTube), The unreasonable effectiveness of BM25 for agentic search](https://youtu.be/fZH97QHHYjY)
- [arXiv, BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://arxiv.org/abs/2508.06600)
- [arXiv, Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?](https://arxiv.org/abs/2605.10848)
- [GitHub, justram/pi-serini](https://github.com/justram/pi-serini)
- [Hugging Face, Tevatron/browsecomp-plus-runs](https://huggingface.co/datasets/Tevatron/browsecomp-plus-runs)
- [arXiv, A Reference Architecture for Agentic Hybrid Retrieval in Dataset Search](https://arxiv.org/abs/2604.16394)
- [Hornet, Deep research is a retrieval problem](https://hornet.dev/blog/deep-research-is-a-retrieval-problem)
- [Hornet, This is what agentic retrieval looks like](https://hornet.dev/blog/this-is-what-agentic-retrieval-looks-like)
- [Hornet, Hybrid search depends on query type](https://hornet.dev/blog/100m-doc-search-part-3-hybrid-search)
- [Wikipedia, Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25)