crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Findings

What the index actually shows

Four things stand out from 3,656 measured domains, and three of them are not what the category usually reports. Every figure here recomputes from the published dataset, so it moves when the web does.

Finding one

Thousands of sites enforce a policy they never published

robots.txt is a published promise. What a server does when a crawler carrying an AI user agent actually knocks is a separate fact, and the two do not have to agree. Every other index in this category publishes the first one. This index has measured both on every domain since the first crawl.

571 measured sites, or 15.6%, permit GPTBot in robots.txt and refuse a request from GPTBot at the server. Nothing in their published policy asked for that. It is almost always an edge rule switched on above the operator, and the operator usually does not know.

The measurement is deliberately narrow: only an outright refusal counts, never a response that merely looks thin, because a dynamic page varies legitimately and a false accusation here would be expensive.

Sites grouped by what robots.txt states against what the server does when asked as GPTBot
GroupSitesShare
Says yes, does no57115.6%
Open, and means it255269.8%
Closed, and means it2216.0%
Says no, does yes1764.8%

The most-visited sites where this is happening

Sites permitting GPTBot in robots.txt and refusing it at the server
RankDomainScoreAnswer-surface crawlersStack
24appsflyersdk.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedAmazon CloudFront
57cloudflare.netrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
80epicgames.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
100wordpress.comrefused GPTBotScore 77 out of 100, grade Ball allowedWordPress
130nih.govrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
146okcdn.rurefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedunidentified
153tumblr.comScore 54 out of 100, grade D4 blockedWordPress
173amazon-adsystem.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedunidentified
175paypal.comrefused GPTBotScore 92 out of 100, grade A, partial assessmentall allowedFastly
208skyhigh.cloudrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
221charter.netrefused GPTBotScore 45 out of 100, grade Dall allowedAmazon CloudFront
229namecheap.comrefused GPTBotScore 92 out of 100, grade A, partial assessmentall allowedCloudflare
231sciencedirect.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
239researchgate.netrefused GPTBotScore 78 out of 100, grade B, partial assessment1 blockedCloudflare
311springer.comScore 92 out of 100, grade A, partial assessmentall allowedFastly
321wiley.comrefused GPTBotScore 47 out of 100, grade Dall allowedCloudflare
345etsy.comrefused GPTBotScore 84 out of 100, grade B, partial assessmentall allowedFastly
362gnu.orgrefused GPTBotScore 54 out of 100, grade Dall allowedunidentified
367checkpoint.comrefused GPTBotScore 70 out of 100, grade Call allowedWordPress . Amazon CloudFront
374un.orgrefused GPTBotScore 45 out of 100, grade Dall allowedDrupal . Amazon CloudFront
378browser-intake-datadoghq.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedunidentified
400cookiedatabase.orgrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
428163.comrefused GPTBotScore 51 out of 100, grade Dall allowedAlibaba Cloud CDN
437indeed.comrefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedCloudflare
441telekom.derefused GPTBotScore 78 out of 100, grade B, partial assessmentall allowedunidentified

Finding two

Most of the web never made a decision about AI at all

Coverage of AI crawler blocking treats it as a choice publishers are making. For most of the web it is not a choice, it is a default.

1,716 measured sites have a robots.txt that names no AI crawler at all, and another 969 have no robots.txt whatsoever. Against that, only 816 name a single one. So for roughly 73.4% of the sites in this index, whatever AI policy exists arrived with the platform or the CDN.

Policy postureWhether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. More

Share of measured sites in each policy posture
GroupValue (%)
Inherited46.9
Absent26.5
Deliberate22.3
Blanket4.2
Inherited
robots.txt exists and names no AI crawler at all. Whatever AI policy this site has is a side effect of generic rules it inherited, most often from its platform or CDN default.
Absent
No robots.txt. Crawler policy is undeclared, so every crawler applies its own default.
Deliberate
robots.txt names AI crawlers by token. Somebody at this organisation decided what AI systems may do with the site.
Blanket
One rule for every crawler, allow nothing. This is a decision, but it is not a decision about AI specifically.

Blocking rate by
edge networkThe CDN or reverse proxy in front of the origin: Cloudflare, Akamai, Fastly, CloudFront. It can block a crawler before the site ever sees the request, which is why blocking correlates better with the CDN than with anything the operator published. More

Share of sites behind each edge network blocking at least one answer-surface crawler
GroupValue (%)
cloudflare15.2
cloudfront19.4
akamai14.0
fastly26.8
google2.7
vercel3.2
varnish29.4
alibaba6.5
azure-frontdoor3.6

The spread between the most and least restrictive edge networks is far wider than anything the sites themselves publish accounts for. Which CDN a site sits behind predicts its AI policy better than anything about the site.

Finding three

Blocking is not one behaviour, and the interesting group wants to be cited

A single "blocks AI" percentage flattens six different positions into one. Separating them turns up a segment that is usually invisible: sites that block the training crawlers and allow every crawler that answers a live question. They are not refusing AI. They are refusing to be absorbed while staying available to be quoted.

Open

56.0%

2,046 sites

Every answer-surface crawler is allowed. An agent asked about this site can read it.

Undeclared

26.5%

969 sites

No robots.txt, so nothing is stated either way.

No training

6.6%

241 sites

Training crawlers are blocked and the crawlers that answer live questions are not. This site wants to be cited without being absorbed.

Selective

5.2%

189 sites

Some answer-surface crawlers are blocked and others are not.

Walled

4.5%

165 sites

Every answer-surface crawler is blocked. This site is invisible to AI answers by choice.

Assistant only

1.0%

37 sites

Index builders are blocked but crawlers fetching a page on behalf of a person are allowed. Readable on request, not in bulk.

Metered

0.2%

9 sites

Access is sold rather than refused. An unpaid agent gets HTTP 402 Payment Required.

Measured across 11 answer-surface crawlers. Per-crawler detail.

Finding four

Almost nobody publishes anything for agents to read

There is a lot of writing about llms.txt, agent cards and machine-readable licensing. There is very little deployment. These are the adoption rates across the most-visited domains on the web, which is the most favourable population you could pick.

llms.txt559 of 3,656 sites (15.3%)
agents.md31 of 3,656 sites (0.8%)
Agent card10 of 3,656 sites (0.3%)
Machine-readable licence13 of 3,656 sites (0.4%)
Content-Signal163 of 3,656 sites (4.5%)
A dateline489 of 3,656 sites (13.4%)

Declared authorship reaches 8.7%. An answer engine deciding whether to quote a page weighs when it was written and who wrote it, and both are cheaper to add than anything else on this list.

By stack

Cohorts under 25 measured sites are not published, because a 100% blocking rate over three sites is noise presented as a finding.

Edge network

Blocking rate by edge network
Edge networkSitesBlocking AIProportion blockingMean score
Cloudflare1,127171 (15.2%)70.1
Amazon CloudFront44386 (19.4%)64.3
Akamai28540 (14.0%)68.9
Fastly21357 (26.8%)63.7
Google Cloud / GFE1484 (2.7%)58.3
Vercel622 (3.2%)76.3
Varnish3410 (29.4%)61.6
Alibaba Cloud CDN312 (6.5%)64
Azure Front Door281 (3.6%)62.2

Publishing platform

Blocking rate by publishing platform
PlatformSitesBlocking AIProportion blockingMean score
Next.js32454 (16.7%)67.4
WordPress30252 (17.2%)70.3
Adobe Experience Manager1356 (4.4%)65.2
Drupal1125 (4.5%)63.9
HubSpot CMS823 (3.7%)72.6
Contentful563 (5.4%)69
Nuxt531 (1.9%)65.8
Sanity293 (10.3%)77.5

Top-level domain

Blocking rate by top-level domain
Top-level domainSitesBlocking AIProportion blockingMean score
.com1,970361 (18.3%)66
.org22631 (13.7%)65.2
.net18333 (18.0%)61.4
.ru11512 (10.4%)64.4
.edu826 (7.3%)65.3
.de8021 (26.3%)62
.io786 (7.7%)67.7
.gov714 (5.6%)65.8
.jp343 (8.8%)61.5
.fr3314 (42.4%)57.7

Check any of this yourself

Every number on this page is arithmetic over an archived observation, no model involved at any point, and the whole dataset is a download. How each one is measured . What the index does not cover . Download it

crawl 2026-11-22 10:38 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,656 of 5,006 reachable

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "What the CrawlIndex data shows." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.