Chunking is not dead. The talk that says so smooths one of its own numbers.

In an AI Engineer conference talk posted this week, Yuval Belfer of AI21 Labs argued that retrieval-augmented generation has a quieter problem than the RAG-is-dead narrative suggests: nobody has settled on the right chunk size, because the right chunk size depends on the query. His fix, index the same corpus at several chunk sizes at once and fuse the rankings, traces to a real AI21 research post and a real, working method. The specific numbers spoken in the talk, though, round a more skewed underlying result into something that sounds more uniformly impressive than the primary source reports. And a separate, independent 2026 benchmark found the opposite lesson winning on a different corpus: the cheapest, simplest chunking strategy beating every more elaborate one it tested.

#What the talk gets right

The mechanism is real, published, and code-backed. AI21's own blog post on query-dependent chunking describes indexing a corpus at multiple chunk sizes (the post uses 100, 200, and 500 tokens as an example) and combining the separate result sets with Reciprocal Rank Fusion, a rank-aggregation method that needs no score calibration across the different indices and no model retraining. The linked open-source code, AI21Labs/multi-window-chunk-size, is a real, public repository created in January 2026. RRF itself is not AI21's invention. It comes from a peer-reviewed 2009 SIGIR paper by Cormack, Clarke, and Buettcher, a seventeen-year-old technique being pointed at a new problem rather than a novel algorithm dressed up as one.

The oracle-headroom claim also checks out. AI21's post reports oracle experiments showing 20 to 40 percent headroom over any single fixed chunk size, at a real, quantified storage cost of roughly two to five times, since the corpus now has to be indexed at every chunk size in use. That is a genuine, well-scoped tradeoff, not a hand-wave.

#Where the talk's own numbers get rounded

The talk states its MTEB benchmark improvement as "somewhere between 10 to 40% depending on the dataset." AI21's own blog post, the primary source the talk is built on, reports something more specific and considerably more skewed: gains of 1 to 3 percent across seven of the eight benchmark configurations tested, with one outlier, TRECCOVID on the E5-small model, showing a 36.7 percent jump. Technically, 36.7 falls inside "up to 40%." But "10 to 40% depending on the dataset" reads to a listener as a range of ordinary outcomes, when the actual shape is typically low single digits, with one large exception. That is not a fabrication. It is a compression of a skewed distribution into language that implies a smoother, more uniformly favorable result than the underlying data shows.

A second claim does not check out as sourced. The talk attributes a "chunking is dead" declaration to Jerry Liu, CEO of LlamaIndex, as color commentary for why the field supposedly needs an even better answer than agentic search. No sourced quote for that attribution exists. The real LlamaIndex post on this subject is titled "Naive RAG is dead, agentic retrieval is the future," and its argument is scoped to naive top-k retrieval generally, not chunking specifically. Notably, that post's own most advanced retrieval mode retrieves whole files rather than chunks at all, which if anything pushes further away from chunking than the "chunking is dead" framing implies, not less far. The rhetorical move (a real hype cycle exists, so cite a specific person as having said the extreme version of it) does not survive a direct check here.

#What the talk does not mention

The sharper complication comes from outside the talk entirely. A large, independent 2026 benchmark (Vecta/FloTorch), summarized by premai.io, tested seven chunking strategies across 50 academic papers, more than 900,000 tokens of text. The winner was recursive character splitting at 512 tokens with 50 to 100 tokens of overlap: the simplest, cheapest strategy on the list, requiring zero model calls, and it outperformed every more elaborate alternative tested in that specific benchmark. The same source cites a separate finding from Chroma's own chunking evaluation showing only a 9 percent recall gap between the best and worst chunking strategy on one corpus, far smaller than AI21's cited 20 to 40 percent oracle headroom. Different corpora and different metrics (recall at some fixed K versus end-to-end answer accuracy) explain part of that gap, but the honest reading is that "how much does chunk size actually matter" does not have one settled answer across the field yet.

A separate empirical study from PwC researchers, "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search," complicates the talk's other assumption, that plain agentic tools like grep, ls, and find are not sophisticated enough to replace retrieval tuning. That paper found grep-based lexical retrieval generally outperforming vector retrieval across several agent harnesses, including Claude Code, Codex, and Gemini CLI, in its own comparisons. That study measured a memory and conversation-history retrieval task, not the document-QA setting AI21's benchmarks target, so the two results are not directly contradictory. They simply do not point in the same direction, and the talk does not address that its own dismissal of grep-style tools sits opposite a real, dated 2026 finding.

#What an operator does with this

The mechanism AI21 built is genuinely useful and genuinely novel in application: chunk size is measurably query-dependent, not just intuitively so, and a 17-year-old rank-fusion method recovers a meaningful slice of that headroom for a real, quantified cost. That part of the talk is worth taking seriously.

What is not worth taking as settled is the size and universality of the benefit. One 2026 benchmark found multi-scale indexing worth its storage cost. A separate, larger 2026 benchmark on a different corpus found the plainest, cheapest fixed-chunking strategy beating everything more elaborate. A third study found grep beating vector retrieval on a different task entirely. None of these results overturn each other, because none of them were run on the same corpus against the same task. The actual, harder-won lesson across all three is that the winning retrieval strategy is corpus- and task-dependent, and no single talk's headline number, including this one's, is a substitute for measuring your own data before committing to a specific chunking or retrieval architecture. A talk with real, sourced numbers and a working method can still oversell its own scope through a smoothed statistic and an unsourced line about a competitor. Check the primary source before repeating either.