ena pragma
·7 min read·geo · technical-seo · ai-crawlers · methodology

How to measure what AI crawlers actually see on your site

A method you can run in an afternoon, with our own numbers. Fetch as GPTBot, diff the bytes, make MISSING a real result.

Contents

Most advice about AI crawlers is about what to put on your site. This is about how to find out what your site is actually serving them, which is a different question and a cheaper one to answer.

You can run all of it in an afternoon with a scripting language and no paid tools. We ran it on our own site last week and the numbers below are ours, including the parts that came back less useful than we hoped.

#Start with the list, not the homepage

Read your own sitemap.xml and use it as the population. Checking the homepage tells you about the homepage.

Ours returned 130 URLs. Fetching all of them took a few minutes and produced zero fetch errors, which is itself a result worth having: every page a crawler is told about is reachable.

If your sitemap and your real page set disagree, stop here and fix that first. Everything downstream inherits the error.

#Record what is served, not what the template intends

For each URL, pull the values out of the served HTML rather than reading the component that is supposed to produce them. Those two things drift, and only one of them is what a crawler gets.

The fields worth capturing per page:

  • <title>
  • meta name="description"
  • link rel="canonical"
  • og:title, og:description, og:image
  • twitter:card
  • every @type inside your JSON-LD blocks

Ours came back: title, description and canonical present on all 130. Open Graph and Twitter card missing on 2. Schema missing on 3.

#Make "missing" a real value, and prove it

This is the part most audits skip, and it is the part that decides whether you can trust anything above.

If your extractor returns an empty string for a field that is absent, then an absent field and a present-but-empty field look the same, and so does a bug in your own regex. Return a distinct MISSING value instead.

Then prove the check can produce it. Build two fixtures: a page with none of the fields, and a page with all of them. Assert that the first reports every field missing and the second reports none.

RED ARM  bare page -> all 8 fields MISSING: True
RED ARM  full page -> no field MISSING    : True

That is thirty seconds of work and it is the difference between "the site is clean" and "my script found nothing, for some reason." Ours passed, so the counts above mean absent rather than unmeasured.

#Fetch as the crawlers, and diff the bytes

Now the part specific to AI answer engines. Request the same URL with different user agents and compare what comes back:

  • your default agent
  • GPTBot
  • ClaudeBot
  • PerplexityBot

Compare the response status, the byte length, and the number of JSON-LD nodes. You are looking for any difference at all.

Ours were byte-identical. All four returned HTTP 200 at 62,335 bytes with the same three schema nodes. Two useful conclusions follow: nothing is gating content by user agent, and the schema is present in the server-rendered HTML rather than injected by JavaScript, so a crawler that does not execute JS still receives it.

If your byte counts differ across agents, you have found something important and it is worth understanding before you change anything.

#Read robots.txt as a position, not a default

Fetch it and read the whole thing rather than checking whether it exists.

The question is not "are we blocking crawlers." It is "did anyone decide this." A file naming fifteen agents individually was written on purpose. A bare User-agent: * / Allow: / may never have been thought about at all.

Ours enumerates 22 named agents and allows every one. That list was deliberate. Whether allowing all of them was deliberate is a separate question, and the two most worth re-examining are the training-corpus crawlers rather than the answer engines, because those are the ones most likely to have been allowed by inclusion rather than by decision.

Also check your response headers for x-robots-tag, not just the file. A header-level directive is invisible to anyone reading robots.txt, and it wins. Ours: absent on HTML pages, so robots.txt is the only thing governing crawl, and no header can silently contradict it.

#Check that schema validates, not just that it parses

Parsing proves the JSON is well formed. It says nothing about whether the required properties are there.

Walk each node and check the fields its type actually needs: Organization and WebSite want name and url; BlogPosting wants headline, datePublished and author; BreadcrumbList and ItemList want itemListElement. Confirm @context is present, because a node without it is decoration.

Ours: zero structural issues across 130 pages, @context everywhere.

Be careful how you read that result. It is a required-field check against the types you serve. It is not a rich-results test and it does not prove any engine will display anything.

#Two things worth measuring while you are in there

Cache age. Record age and your CDN's cache-status header. Ours showed roughly two and a half hours of held content with a cache hit. That single number explains a failure mode that costs teams real time: fetch a page immediately after deploying and you can get the old version, or a 404 for a page that already exists, and nothing in the response tells you that is what happened. Add a cache-busting query parameter to any post-deploy check.

Structure and alt text. Heading hierarchy and image alt attributes, per page. Ours came back with one faulty page out of 130, and every fault on that same page: no h1, three images with no alt, three internal links against a site mean of 26.

That page is an interactive tool listed in the sitemap as though it were a document. Five separate checks converged on one URL, and it was one fact rather than five defects. Worth remembering when a report gives you a cluster: ask whether it is one thing wearing five costumes.

#What this method cannot tell you

State this plainly to yourself before you present results to anyone.

It measures what you emit, not what engines return. A site can serve perfect metadata to every crawler and appear in no answers at all, and everything above would look green. Those are different measurements with different instruments, and the second one is the one your buyers experience.

Schema may not be a ranking lever. Our own read of the research is that structured data helps an engine work out which entity you are rather than lifting citation rates directly. We have not reproduced that ourselves, so treat it as a reason to keep expectations modest rather than as a settled finding. Either way, disambiguation is worth having and is not the same thing as being cited.

It cannot measure page speed. Core Web Vitals need a real browser with rendering and network throttling. HTML weight and asset counts are inputs to speed, not the measurement, and presenting one as the other is worse than reporting nothing.

#The shortest version

  1. Population from your sitemap, not the homepage.
  2. Record what is served, not what the template intends.
  3. Make MISSING a distinct value and prove your check can return it.
  4. Fetch as GPTBot, ClaudeBot and PerplexityBot; diff the bytes.
  5. Read robots.txt as a decision, and check headers too.
  6. Validate required schema fields, not just JSON syntax.
  7. Cache-bust anything you check right after a deploy.
  8. Write down what the method cannot see, next to what it found.

The whole thing is an afternoon. The value is not the green boxes. It is that afterwards you know which of your beliefs about your own site were measured and which were assumed.

#Sources

Keep reading