ena pragma
·9 min read·ai-operations · ai-verification · ai-vendor-selection

AI agents still struggle to read a PDF. The benchmark that shows it was built by the company selling the fix.

Jerry Liu of LlamaIndex argues documents are the missing context layer for AI agents. His own benchmark supports much of it, and it is still a vendor's benchmark.

Contents

A company that wants an AI agent to handle its invoices, insurance claims or customer onboarding runs into the same problem first: the information is locked in PDFs, slide decks, Word files and spreadsheets. A talk by Jerry Liu, co-founder and CEO of LlamaIndex, given at the AI Engineer World's Fair this summer and published on September 23, 2026, argues that turning those files into something an agent can trust is the unsolved part of building agents.

Much of his argument holds up. It is also a pitch: LlamaIndex sells a document parser, and the benchmark he uses to show the problem is one his company built. A buyer should read it both ways.

#Why a PDF is hard for a machine to read

Liu's explanation is the clearest part of the talk. A PDF is designed for printing. Its text is stored as characters placed at coordinates on a page, and its tables are line segments with characters positioned inside the cells. On a multi-column page, as Liu puts it, nothing guarantees that the order text is stored in matches the order a person reads it. So an agent handed a raw PDF gets fragments with positions, not a document with structure, and has to reconstruct what a person sees at a glance.

His larger frame, in his words, is that retrieval-augmented generation in 2026 decomposes into "an agent harness plus a context layer." The agent now decides what to search for, instead of relying on a fixed lookup of the closest matches. The hard, unfinished part is the layer underneath: getting documents into a form the agent can use.

#What his benchmark actually shows

LlamaIndex built a benchmark to measure this, called ParseBench. Its paper, posted in April 2026, describes about 2,000 "human-verified pages from enterprise documents spanning insurance, finance, and government," scored on five dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding (whether each extracted piece can be traced back to its place on the page). The paper's dataset table counts 169,011 deterministic test rules (the site rounds it to 169K), and its data and grading code are public, so anyone can rerun it.

The paper finds that "no method is consistently strong across all five dimensions," and in the same breath names its own LlamaParse Agentic the top overall scorer. The live leaderboard, as of September 23, 2026, ranks 112 methods. In the talk Liu said the benchmark covered about fifty; the count has more than doubled since.

At the top, as of that date, are LlamaIndex's own products: LlamaParse Agentic Plus at 90.2 and LlamaParse Agentic at 87.0. The best general-purpose frontier models sit below them: Anthropic's Fable 5.1 at 78.9 (sixth) and Opus 5.5 at 78.0 (eighth), with OpenAI's GPT-6 Sol (medium reasoning) at 68.2.

The more useful detail is where the frontier models fall short and where they do not.

  • Visual grounding and charts are the gap. Opus 5.5 scores 63.2 on visual grounding, against 84.3 for LlamaParse Agentic, and 64.1 on charts.
  • Tables are not a gap. Opus 5.5 has the highest tables score on the whole board, 93.9.
  • Reading the text is not the gap. On content faithfulness, Opus 5.5 scores 91.8, level with LlamaParse Agentic at 91.8. In the talk, Liu said that sending a page straight to a vision-language model "can hallucinate" (the captions are garbled about which pages). His own leaderboard does not show the top models doing worse than LlamaParse on this dimension.
  • Cost is a real gap. Per page, the leaderboard lists Fable 5.1 at 16.1 cents and Opus 5.5 at 5.8 cents, against 1.25 cents for LlamaParse Agentic and about 0.4 cents for LlamaParse's cost-effective tier. LlamaIndex's top-ranked Agentic Plus costs 5.6 cents, about the same as Opus 5.5. At a million pages a day, these differences are the budget.

#Every vendor wins its own benchmark

ParseBench's authors are LlamaIndex staff, and in the paper's results table its own product is labelled "LlamaParse (Ours)." That does not make the results wrong. It does mean the benchmark measures what LlamaIndex chose to measure. LlamaIndex has also been open that benchmarks shape its output: its LiteParse release post says "tuning output to win one benchmark (e.g. olmOCR-bench) would regress another (e.g. ParseBench)."

One small signal is worth noticing. The entry listed as Docling-models, in the leaderboard's open-weight vision-model category, scores 1.0 out of 100 on semantic formatting while scoring 50.6 overall. Our reading, not the benchmark's: a near-zero on one dimension suggests the scoring rewards a specific output convention that the system does not use, rather than one that cannot parse at all.

Some of the clearest advice on this comes from another vendor. Datalab, which competes in the same market, wrote in July about a benchmark commissioned by a third competitor, Reducto: "Like most vendor benchmarks, it was specifically designed for marketing - Reducto was intended to win," adding, "we'd encourage you not to trust vendor benchmarks (including ours!), and to run your own evals." Datalab now leads on the 50-document public set of that benchmark, and says so in the same post. The advice stands anyway.

#The free parser, and what "fastest" and "most accurate" mean

Liu also promotes LiteParse, a free parser from LlamaIndex. (The talk's official chapter list spells it "LightParse"; the project is LiteParse.) It is released under the Apache 2.0 license and written in Rust.

In the talk he calls it "the fastest open source parser out there." The company's own measurement is narrower. LlamaIndex's release post describes "the fastest open-source, model-free, pdf-to-markdown pipeline," and its speed table shows LiteParse at 3.16 milliseconds per page against 3.83 for the next tool tested. That is a lead of about 17% in a specific category, measured by the vendor.

Liu also called it "the most accurate like markdown parser out there that doesn't use a VLM or any sort of kind of like deeper model." The live leaderboard does not bear that out. As of September 23, 2026, its no-OCR configuration ranks 99th of 112 overall with a score of 36.9, and third of the nine parsers the leaderboard lists as open-source and local, behind PyMuPDF4LLM (53.5) and Warp Ingest (40.2). LlamaIndex's June release post reported LiteParse leading the model-free tools it compared; that post reported LiteParse at 0.3279 on ParseBench where the leaderboard now shows 36.9 out of 100, so the two reflect different runs, and we did not reconcile them. (Its chart score of 1.8 is not the point: LlamaIndex's own post notes that charts are "effectively noise for every model-free tool.") The same post is candid about the ceiling: with this approach, "we aren't going to do better than LlamaParse."

Liu positions it as a fast first pass that an agent runs before sending hard pages to a slower, more capable reader. That is a sensible design. The scores do not support using it alone where accuracy matters.

#The size of the problem

Liu says there are "over 10 trillion plus pages" of knowledge locked in these file types. He gives no source for that figure, and we found none. The closest first-party number we found is Adobe's estimate of "approximately 3 trillion PDFs in the world," which counts files, not pages. The order of magnitude is plausible. The exact number is not established.

#The throughline, as we read it

What follows is our interpretation, not a claim any single source makes.

What. AI agents can now do knowledge work, and most of the knowledge a business has sits in documents made for printing. The layer that turns those documents into something an agent can use and trust sits underneath every agent a company builds.

Who. Parser vendors are racing to own that layer, and they are competing partly on benchmarks they build themselves. LlamaIndex built ParseBench and offers LiteParse free as an on-ramp to its paid LlamaParse.

How. Liu describes three layers: parse the document into structured text, store it so people and agents can find it, and run repeatable workflows on top, such as invoice processing, customer identity checks and insurance claims.

When. Now, because frontier models have become good enough at reading that simply sending the PDF to the model is tempting. The leaderboard shows where that still breaks: grounding, charts, and cost at volume.

Why it matters. An agent's answer is only as trustworthy as the document it read, and visual grounding, one of the two dimensions where frontier models score worst, is exactly the one an audit trail depends on.

#What this means for a business with documents to process

We wrote earlier about what it takes to make a business usable by AI agents, which is mostly about the tools an agent can call. Documents are the other half: what the agent has to read before it can act.

  • Decide which problem you have first. Liu describes three regimes: very high accuracy (insurance and financial services), low cost (indexing millions of documents a day), and low latency (processing uploads in real time). They are different purchases. His framing is useful even though he is selling one answer.
  • Test on your own documents. A vendor benchmark tells you where to look, not what to buy. Take a sample of your real invoices or claim forms, run the two or three leading options, and score them on the fields you actually use. ParseBench's open grading code makes that practical.
  • Pay for grounding, not just extraction. A parser that is 90% right on a claims form is a risk unless every extracted field points back to its place on the page and uncertain fields go to a person. Check that before you check anything else. It is the same principle behind our guide to reading AI claims: a number without a way to check it is a claim, not a result.
  • Use the cheap pass, and do not mistake it for the answer. A free local parser over everything, with an expensive reader only on the hard pages, is a real cost lever. The first pass alone is not good enough for the fields that matter.

How this was researched: our research teammate located the sources using Brave Search and Firecrawl, and we re-read the ParseBench paper, the LiteParse release post, license and README, the Adobe and Datalab pages, and the leaderboard's published data file directly on September 23, 2026. Leaderboard ranks and scores are as of that date and will change. We captured the talk's auto-generated captions ourselves, and quotes from the talk are taken from them.

#Sources

Keep reading