# CrawlIndex

> CrawlIndex measures how thousands of the most-visited websites treat AI crawlers and agents. Which bots they block, whether they publish llms.txt or agents.md, whether they serve crawlers the same content as browsers, and which platforms and CDNs are quietly deciding that policy for them. Updated nightly, method published in full, dataset open.

Research and data by Fidget Labs BV (https://fidgetlabs.io), Breda, Netherlands.
Licensed CC BY 4.0. Attribution to Fidget Labs BV is required when reusing these figures.
Scores are arithmetic over archived evidence. No language model is involved in producing them.

## Current findings (crawl of 2026-08-09, 3,676 domains measured)

- 661 sites block at least one answer-surface AI crawler.
- 158 sites block every answer-surface AI crawler.
- 448 sites publish an llms.txt.
- 27 sites publish an agents.md.
- 952 sites refused or curtailed a request identifying as GPTBot while serving a browser normally. For some, robots.txt already blocks GPTBot, so the server is consistent rather than contradictory. The per-domain endpoint separates the two cases.
- 12 sites answered an agent with HTTP 402 Payment Required, which is a pay-per-crawl gateway rather than a block.
- Mean agent readiness score: 63.82 out of 100.

## Who is setting the policy

Most operators never formed a view on AI crawlers. Their edge network or platform shipped a default and they inherited it. Blocking rate by edge network:

- cloudflare: 17.5% of 1,076 measured sites block an answer-surface crawler.
- cloudfront: 20.5% of 424 measured sites block an answer-surface crawler.
- akamai: 14.3% of 308 measured sites block an answer-surface crawler.
- fastly: 24.9% of 221 measured sites block an answer-surface crawler.
- google: 3.2% of 154 measured sites block an answer-surface crawler.

By publishing platform:

- wordpress: 17.4% of 299 measured sites.
- nextjs: 16.3% of 295 measured sites.
- aem: 3.9% of 155 measured sites.
- drupal: 3.3% of 120 measured sites.
- hubspot: 3.8% of 78 measured sites.

## Start here

- [Methodology](https://crawlindex.org/methodology): what is requested, how robots.txt is interpreted, and the full 100-point rubric.
- [Download the dataset](https://crawlindex.org/data): every record, free, no key, no signup.
- [Leaderboard](https://crawlindex.org/leaderboard): most and least agent-ready sites.
- [Does your CDN decide your AI policy](https://crawlindex.org/networks): blocking rate by edge network.
- [Readiness by platform](https://crawlindex.org/platforms): blocking rate by CMS and framework.
- [By top-level domain](https://crawlindex.org/tlds): a rough proxy for jurisdiction.
- [Crawler registry](https://crawlindex.org/bots): every AI crawler tracked, and how many sites block it.
- [Change feed](https://crawlindex.org/changes): sites that recently changed crawler policy.
- [About](https://crawlindex.org/about): who runs this, how it is funded, and how to be removed.

## Reports

- [The state of AI crawler access, August 2026](https://crawlindex.org/reports/2026-08)

## Machine endpoints

- [Full dataset](https://crawlindex.org/data/domains.jsonl): JSON Lines, one record per domain, with the archived observation behind every score.
- [Daily statistics](https://crawlindex.org/data/stats.json): the complete series since the index began.
- [Change log](https://crawlindex.org/data/changes.jsonl)
- [Aggregate statistics](https://crawlindex.org/api/v1/stats): current totals and cross-tabs as JSON.
- [Single domain](https://crawlindex.org/api/v1/domain/stripe.com): full measurement for one domain.

## Per-crawler pages

- [Who blocks GPTBot](https://crawlindex.org/bots/gptbot): OpenAI. Collects training data for OpenAI models.
- [Who blocks OAI-SearchBot](https://crawlindex.org/bots/oai-searchbot): OpenAI. Builds the index behind ChatGPT search results.
- [Who blocks ChatGPT-User](https://crawlindex.org/bots/chatgpt-user): OpenAI. Fetches a page live when a ChatGPT user asks about it.
- [Who blocks ClaudeBot](https://crawlindex.org/bots/claudebot): Anthropic. Collects training data for Anthropic models.
- [Who blocks Claude-User](https://crawlindex.org/bots/claude-user): Anthropic. Fetches a page live on behalf of a Claude user.
- [Who blocks Claude-SearchBot](https://crawlindex.org/bots/claude-searchbot): Anthropic. Builds the index behind Claude search results.
- [Who blocks PerplexityBot](https://crawlindex.org/bots/perplexitybot): Perplexity. Builds the Perplexity answer index.
- [Who blocks Perplexity-User](https://crawlindex.org/bots/perplexity-user): Perplexity. Fetches a page live for a Perplexity user query.
- [Who blocks Google-Extended](https://crawlindex.org/bots/google-extended): Google. Controls use in Gemini training and grounding. Does not affect Google Search ranking.
- [Who blocks Applebot-Extended](https://crawlindex.org/bots/applebot-extended): Apple. Controls use in Apple Intelligence training.
- [Who blocks meta-externalagent](https://crawlindex.org/bots/meta-externalagent): Meta. Collects training data for Meta AI.

## Optional

- [Score badge](https://crawlindex.org/badge/stripe.com.svg): embeddable SVG for any indexed domain.
- [Source code and full history](https://github.com/kerriganbaron-fidgetlabs/crawlindex)