How the numbers are made

What we request

Each measured domain receives at most five requests, identifying as CrawlIndexBot/1.0 except where noted:

1. /robots.txt first, which yields the access policy for every tracked crawler in one fetch and lets us honour an opt-out before requesting anything else.
2. The homepage, as a mainstream desktop browser. This is the control.
3. The homepage again, as GPTBot, to compare against the control.
4. /llms.txt
5. /agents.md

A site whose robots.txt bars all crawlers costs one request, not five. Requests are rate limited to one per host at a time, carry a From header, and time out quickly. We read only what a site serves publicly. We do not log in, do not evaluate JavaScript, and store no personal data.

What is derived without extra requests

Publishing platform, edge network, server software, page language, feed and canonical presence, security headers and the sophistication of the robots.txt policy are all read from bytes already fetched. That costs the measured site nothing and is what lets this index cross-tabulate blocking against CDN and platform.

29 platform fingerprints and 17 network fingerprints are matched, header evidence before markup, most specific first. Nothing is guessed: an unrecognised stack is recorded as unidentified and excluded from cross-tabs, because a wrong label is worse than no label.

How robots.txt is read

Following RFC 9309. Consecutive User-agent lines share a group. A group naming a token beats the wildcard group outright and does not inherit from it. The longest matching path rule wins, and an exact-length tie goes to Allow. An empty Disallow: means allow everything. Where no rule applies, the crawler is allowed.

Five rules that keep this honest

1. No model touches a score. Scoring is arithmetic over recorded observations. Language models are used nowhere in measurement, scoring or report prose.
2. Unobservable is not zero. A site we cannot reach has no score and is excluded from every average, rather than counted as a failure and dragging the aggregate down.
3. Our failures are not charged to the site. When our control request is met by a bot challenge, everything we could infer from that page describes the challenge, not the site. Those checks are marked unavailable and the score is renormalised over what remained. Such results are labelled partial and kept out of leaderboards, because a score over 46 points is not comparable to one over 100.
4. A change must be the site's, not ours. Nothing is reported as a change when the two observations came from different probe versions or different network vantage points. Improving our own bot-wall detection, or moving where the crawler runs, must never be published as somebody else changing their policy.
5. Everything is versioned. Each stored measurement records the crawler registry, probe and rubric versions and the vantage it was produced under, and the full observation is archived so any score can be recomputed later.

What a low score does not mean

Blocking AI crawlers is a legitimate choice, and for many publishers it is the correct one. A low score means a site is hard for agents to read. It does not mean the site is badly run, and this index takes no position on whether any given operator should open up. What it does insist on is that the choice be visible.