Updated nightly. Method published in full.
The open index of how the web treats AI agents.
We measure the most-visited sites on the web and publish what we find. Which AI crawlers each site blocks. Whether it publishes llms.txt or agents.md. Whether it quietly serves crawlers something different from what it serves you. Every number is recomputable from archived evidence.
What the last crawl found
3,654 domains measured on 2026-08-09. Sites we could not reach are excluded rather than counted as failures.
Which crawlers get shut out
Share of measured sites whose robots.txt denies each crawler the site root.
| Crawler | Operator | Blocked by | Proportion |
|---|---|---|---|
| CCBot | Common Crawl | 589 (16.1%) | |
| GPTBot | OpenAI | 554 (15.2%) | |
| Bytespider | ByteDance | 552 (15.1%) | |
| ClaudeBot | Anthropic | 534 (14.6%) | |
| meta-externalagent | Meta | 495 (13.5%) | |
| Google-Extended | 482 (13.2%) | ||
| Applebot-Extended | Apple | 460 (12.6%) | |
| Amazonbot | Amazon | 449 (12.3%) | |
| Diffbot | Diffbot | 389 (10.6%) | |
| cohere-ai | Cohere | 388 (10.6%) | |
| PerplexityBot | Perplexity | 386 (10.6%) | |
| YouBot | You.com | 333 (9.1%) |
Least agent-ready right now
Fully measured sites with the lowest scores. Partial assessments are excluded because a renormalised score is not comparable to a complete one.
| Rank | Domain | Score | Answer-surface crawlers | Agent files |
|---|---|---|---|---|
| 56 | tiktok.com | Score 15 out of 100, grade F | 11 blocked | none |
| 25 | amazon.com | Score 18 out of 100, grade F | 10 blocked | none |
| 63 | msn.com | Score 18 out of 100, grade F | 11 blocked | none |
| 317 | amazon.co.uk | Score 18 out of 100, grade F | 10 blocked | none |
| 382 | amazon.de | Score 18 out of 100, grade F | 10 blocked | none |
| 501 | amazon.co.jp | Score 18 out of 100, grade F | 10 blocked | none |
| 612 | amazon.fr | Score 18 out of 100, grade F | 10 blocked | none |
| 642 | amazon.ca | Score 18 out of 100, grade F | 10 blocked | none |
| 666 | amazon.in | Score 18 out of 100, grade F | 10 blocked | none |
| 707 | amazon.co.za | Score 18 out of 100, grade F | 10 blocked | none |
Recent movements
Only real changes are recorded. A site's first measurement is a baseline, not an event.
- gap.comScore rose 15 points to 79.
- futbin.comScore fell 17 points to 83.
- furaffinity.netScore fell 4 points to 7.
- ftc.govScore rose 7 points to 62.
- fubo.tvScore rose 14 points to 67.
- frdmhere.comCame back online.
- frontiernet.netWent unreachable: homepage returned HTTP 404.
- franceinfo.frScore rose 7 points to 35.
Why this exists
Publishers are deciding, one robots.txt at a time, whether AI systems may read the web. Those decisions are made quietly, changed without announcement, and are individually trivial to check but collectively invisible.
CrawlIndex checks them on a schedule and keeps the receipts. The scoring rubric is published, every score is arithmetic over archived evidence, and no language model touches the numbers. If you disagree with a result you can read exactly how it was reached.