Recrawled every night. Method published in full. Dataset open.
Most of the web never decided how AI may read it.
Someone decided anyway. We measure the most-visited sites on the internet every night and publish exactly what we find: which AI crawlers each one blocks, what it publishes for agents to read, whether it serves a crawler something different from what it serves you, and who actually made that call.
What we foundMeasure your siteBrowse the index
What the last crawl found
3,642 domains measured on 2026-11-07. Sites we could not observe are excluded rather than counted as failures, which is why these denominators are smaller than the corpus. See the whole funnel.
Block a crawler that answers questions
A crawler whose output reaches a person as an answer today: the ones behind ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and Meta AI. Blocking one has an immediate, visible cost. MoreSay one thing and do another
robots.txt permits GPTBot and the server refuses GPTBot anyway. The operator published one policy and a different one is being enforced, almost always by an edge rule switched on above them. MorePublish an llms.txt
A markdown file at the site root that points an AI agent at the pages worth reading. A community convention rather than a ratified standard, and adoption is still small. MoreMean readiness score
Zero to one hundred, from three bands: whether AI crawlers are allowed at all (45 points), whether the site publishes machine-readable surfaces (25), and whether its content is structured enough to be read (30). Arithmetic over archived evidence. No model is involved. MoreThe gap between what sites say and what they do
robots.txt is a published promise. What a server does when an AI crawler actually knocks is a separate fact. Every index in this category publishes the first one. We measure both on every domain, and 573 sites turn out to be enforcing a policy they never published.
| Group | Sites | Share |
|---|---|---|
| Says yes, does no | 573 | 15.7% |
| Open, and means it | 2538 | 69.7% |
| Closed, and means it | 221 | 6.1% |
| Says no, does yes | 177 | 4.9% |
Who is actually deciding
1,774 measured sites have a robots.txt that names no AI crawler at all, against 745 that name at least one. For most of the web, the AI policy is a side effect of a default somebody else shipped. Sites behind fastly block an answer-surface crawler 24.8% of the time against 2.5% behind google, a spread far wider than anything the sites themselves publish explains.
Policy posture
Whether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. More
Policy posture
Whether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. More- Inherited1,774 (49%)
- Absent973 (27%)
- Deliberate745 (20%)
- Blanket150 (4%)
Blocking rate by edge network
The CDN or reverse proxy in front of the origin: Cloudflare, Akamai, Fastly, CloudFront. It can block a crawler before the site ever sees the request, which is why blocking correlates better with the CDN than with anything the operator published. More
edge network
The CDN or reverse proxy in front of the origin: Cloudflare, Akamai, Fastly, CloudFront. It can block a crawler before the site ever sees the request, which is why blocking correlates better with the CDN than with anything the operator published. More| Group | Value (%) |
|---|---|
| cloudflare | 15.9 |
| cloudfront | 19.5 |
| akamai | 15.1 |
| fastly | 24.8 |
| 2.5 | |
| vercel | 3.4 |
| varnish | 24.2 |
| alibaba | 6.3 |
All edge networks . By publishing platform . The full argument
Readiness by platform
What a site is built on predicts how legible it is to an agent, because the defaults come with the box.
| Platform | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| Next.js | 311 | 52 (16.7%) | 66.5 | |
| WordPress | 307 | 51 (16.6%) | 70.3 | |
| Adobe Experience Manager | 137 | 9 (6.6%) | 64.5 | |
| Drupal | 109 | 5 (4.6%) | 64.1 | |
| HubSpot CMS | 82 | 4 (4.9%) | 72.7 | |
| Contentful | 59 | 3 (5.1%) | 68 |
Which crawlers get shut out
Share of measured sites whose robots.txt denies each crawler the site root. The first 11 are answer-surface crawlers
A crawler whose output reaches a person as an answer today: the ones behind ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and Meta AI. Blocking one has an immediate, visible cost. More
| Crawler | Operator | Blocked by | Proportion |
|---|---|---|---|
| CCBot | Common Crawl | 563 (15.5%) | |
| Bytespider | ByteDance | 549 (15.1%) | |
| GPTBot | OpenAI | 531 (14.6%) | |
| ClaudeBot | Anthropic | 509 (14.0%) | |
| meta-externalagent | Meta | 484 (13.3%) | |
| Diffbot | Diffbot | 461 (12.7%) | |
| Google-Extended | 460 (12.6%) | ||
| Applebot-Extended | Apple | 448 (12.3%) | |
| cohere-ai | Cohere | 447 (12.3%) | |
| Amazonbot | Amazon | 437 (12.0%) | |
| PerplexityBot | Perplexity | 349 (9.6%) | |
| YouBot | You.com | 329 (9.0%) |
How the web scores
Every fully measured site, in ten-point bands, tinted by the grade each band falls under. The shape is the finding: 74% of the web sits between 50 and 79. Not hostile to agents, not ready for them either. Hover or tab through a band to see what is in it.
| Score band | Grade | Sites | Share |
|---|---|---|---|
| 0 to 9 | F | 0 | 0.0% |
| 10 to 19 | F | 5 | 0.2% |
| 20 to 29 | F | 17 | 0.7% |
| 30 to 39 | F | 52 | 2.0% |
| 40 to 49 | D | 283 | 10.8% |
| 50 to 59 | D | 694 | 26.5% |
| 60 to 69 | C | 767 | 29.3% |
| 70 to 79 | C and B | 463 | 17.7% |
| 80 to 89 | B | 312 | 11.9% |
| 90 to 100 | A | 21 | 0.8% |
Scores 0 to 9 . grade F
0 sites, 0.0% of the indexBlocks most answer-surface crawlers, or serves almost nothing a crawler can read.
No measured site currently scores in this band.
Scores 10 to 19 . grade F
5 sites, 0.2% of the indexBlocks most answer-surface crawlers, or serves almost nothing a crawler can read.
For exampleamazon.com 15tiktok.com 15amazon.ca 15
Scores 20 to 29 . grade F
17 sites, 0.7% of the indexBlocks most answer-surface crawlers, or serves almost nothing a crawler can read.
For examplelaunchpad.net 20cnbc.com 24usatoday.com 24
Scores 30 to 39 . grade F
52 sites, 2.0% of the indexBlocks most answer-surface crawlers, or serves almost nothing a crawler can read.
For examplecnn.com 31bbc.com 31bbc.co.uk 34
Scores 40 to 49 . grade D
283 sites, 10.8% of the indexSubstantially closed or substantially unreadable. Often a platform default rather than a decision.
For exampleyahoo.com 43amazonvideo.com 40yandex.net 49
Scores 50 to 59 . grade D
694 sites, 26.5% of the indexSubstantially closed or substantially unreadable. Often a platform default rather than a decision.
Scores 60 to 69 . grade C
767 sites, 29.3% of the indexReadable but undeclared. Typically no llms.txt, thin structured data, and a robots.txt that names no AI crawler.
For examplemicrosoft.com 62wikipedia.org 66cloud.microsoft 60
Scores 70 to 79 . grades C and B
463 sites, 17.7% of the indexReadable but undeclared. Typically no llms.txt, thin structured data, and a robots.txt that names no AI crawler.
For exampleapple.com 77github.com 72gandi.net 77
Scores 80 to 89 . grade B
312 sites, 11.9% of the indexBroadly legible to agents with room left on the table. Usually one or two changes short of an A.
Scores 90 to 100 . grade A
21 sites, 0.8% of the indexOpen to the crawlers that answer questions, publishing machine-readable surfaces, and serving content an agent can read without running JavaScript.
For exampleappsflyer.com 95atlassian.com 91usercentrics.eu 91
Least agent-ready right now
Fully measured sites with the lowest scores, one row per operator. Partial assessments
Some checks could not be observed, usually because a bot wall answered instead of the site, so those points were removed from the total rather than failed. The remaining points are renormalised to one hundred. A partial score is not comparable with a complete one, which is why partial sites are kept out of ranked lists. More
| Rank | Domain | Score | Policy | Stack |
|---|---|---|---|---|
| 31 | amazon.comand 3 more domains with the same policy | Score 15 out of 100, grade F | SelectiveDeliberate | Amazon CloudFront |
| 63 | tiktok.com | Score 15 out of 100, grade F | WalledDeliberate | unidentified |
| 284 | launchpad.net | Score 20 out of 100, grade F | SelectiveDeliberate | unidentified |
| 4849 | gentoo.org | Score 21 out of 100, grade F | WalledDeliberate | Fastly |
| 481 | cnbc.com | Score 24 out of 100, grade F | WalledDeliberate | Akamai |
| 622 | usatoday.com | Score 24 out of 100, grade F | WalledDeliberate | Sitecore |
| 1374 | dw.com | Score 24 out of 100, grade F | SelectiveDeliberate | Akamai |
| 1630 | wnacg.com | Score 24 out of 100, grade F | Assistant onlyDeliberate | Cloudflare |
| 3609 | politico.eu | Score 24 out of 100, grade F | WalledDeliberate | WordPress . Cloudflare |
| 1186 | theconversation.com | Score 26 out of 100, grade F | WalledDeliberate | Next.js . Fastly |
Recent movements
Only real changes are recorded. A site's first measurement is a baseline, not an event, and nothing is diffed across a change to our own probe.
- zillow.comScore fell 28 points to 50.
- zemanta.comScore fell 7 points to 67.
- ys7.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
- youku.comScore rose 26 points to 78.
- wikihow.comScore fell 10 points to 21.
- vungle.comScore fell 7 points to 69.
The short answers
Every figure on this page with its denominator and its measurement date attached, so quoting one correctly takes no work. Free to reuse under CC BY 4.0 with credit.
- What percentage of websites block AI crawlers?
- 17.6% of measured sites block at least one AI crawler that answers questions today, 642 of 3,642 domains. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 17.6%
- How many websites block every AI crawler?
- 4.3% of measured sites block every answer-surface AI crawler, 156 of 3,642 domains. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 4.3%
- Do websites enforce the AI crawler policy they publish?
- Often not. 15.7% of measured sites, 573 of 3,642, permit GPTBot in robots.txt and then refuse a request from GPTBot at the server. The published policy and the enforced one disagree, usually because of an edge rule applied above the site operator. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 15.7%
- How many websites publish an llms.txt file?
- 15% of measured sites publish an llms.txt, 547 of 3,642 domains. Adoption remains very small even across the most-visited sites on the web. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 15%
- How many websites publish an agents.md file?
- 0.9% of measured sites publish an agents.md, 31 of 3,642 domains. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 0.9%
- Do website owners actually choose their AI crawler policy?
- Mostly not. 75.4% of measured sites, 2,747 of 3,642, name no AI crawler in robots.txt at all, either because the file names none or because there is no file. Only 745 name one explicitly. For most of the web the AI policy arrived as a platform or CDN default rather than as a decision. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 75.4%
- Are any websites charging AI crawlers for access?
- Yes, but very few. 9 of 3,642 measured sites answer an unpaid agent with HTTP 402 Payment Required, metering access rather than refusing it. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 0.2%
- How ready is the average website for AI agents?
- The mean agent readiness score is 64.04 out of 100 across 3,642 measured domains. Measured 2026-11-07 by Fidget Labs BV and published as CrawlIndex. 64.04
Why this exists
Publishers are deciding, one robots.txt at a time, whether AI systems may read the web. Those decisions are made quietly, changed without announcement, and are individually trivial to check but collectively invisible.
CrawlIndex checks them on a schedule and keeps the receipts. The rubric is published, every score is arithmetic over archived evidence, and no language model touches the numbers. The whole dataset is downloadable. If you disagree with a result you can read exactly how it was reached and recompute it yourself.
Read the methodologyWhat every term meansDownload the datasetAdd a domain
crawl 2026-11-07 09:39 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,642 of 5,006 reachable
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "The state of AI crawler access." https://crawlindex.org (measured 2026-11-07). Licensed CC BY 4.0.