crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Report . September 2026

The state of AI crawler access, September 2026

Drawn from 20 daily crawls of up to 3,714 Tranco-ranked domains, of which 3,618 could be measured.

Month in progress. These figures move with every nightly crawl until September 2026 ends, at which point the report is written to a permanent file and stops changing. Quote it with the measurement date attached, or wait for the seal.

Research and data by Fidget Labs BV, Breda, Netherlands. Published as CrawlIndex and free to reuse under CC BY 4.0 with attribution.

15.7%
Block at least one answer-surface crawler
567 sites
4.1%
Block every answer-surface crawler
148 sites
13.4%
Publish an llms.txt
484 sites
64
Mean readiness score
Out of 100

What moved

Between 2026-09-01 and 2026-09-22, the number of measured sites blocking at least one answer-surface AI crawler fell by 98 sites. Sites publishing an llms.txt rose by 15 sites, and sites publishing an agents.md fell by 5 sites.

The mean readiness score across scored sites rose 0.72 points to 64.26.

10 sites answered an agent with HTTP 402 Payment Required. That is a pay-per-crawl gateway rather than a block: access is being metered and sold rather than refused.

Sites that could not be measured are excluded rather than counted as failures, so these totals are over the 3,618 domains actually observed. Method: /methodology. Coverage and its limits: /coverage.

Across the month

Blocking an answer-surface crawler665 to 567 over 20 crawls
Publishing llms.txt469 to 484 over 20 crawls
Mean readiness score63.54 to 64.26 over 20 crawls

Who set the policy

Blocking rate grouped by edge network. Sites behind varnish blocked at 25.7% against 3.7% behind azure-frontdoor, which is a far wider spread than anything explained by what the sites themselves publish.

Blocking rate by edge network
Edge networkSitesBlocking AIProportion blockingMean score
Cloudflare1,132114 (10.1%)70.4
Amazon CloudFront43291 (21.1%)64.5
Akamai26943 (16.0%)67.8
Fastly20549 (23.9%)64.5
Google Cloud / GFE1566 (3.8%)57.9
Vercel503 (6.0%)73.6
Varnish359 (25.7%)61.6
Azure Front Door271 (3.7%)60.1

By publishing platform

Blocking rate by publishing platform
PlatformSitesBlocking AIProportion blockingMean score
WordPress30140 (13.3%)70.7
Next.js29148 (16.5%)66
Adobe Experience Manager1275 (3.9%)65.9
Drupal1053 (2.9%)63.3
HubSpot CMS825 (6.1%)71.3
Contentful532 (3.8%)66.9
Nuxt502 (4.0%)62.8
Sanity283 (10.7%)73.8

Blocking by crawler

AI crawlers by number of sites blocking them in September 2026
CrawlerOperatorBlocked byShareOver period
CCBotCommon Crawl48913.5%-102
BytespiderByteDance46012.7%-98
GPTBotOpenAI45712.6%-103
ClaudeBotAnthropic43412.0%-100
meta-externalagentMeta39711.0%-105
Google-ExtendedGoogle38410.6%-104
DiffbotDiffbot37110.3%-6
cohere-aiCohere37010.2%no change
Applebot-ExtendedApple36410.1%-102
PerplexityBotPerplexity3549.8%-8
AmazonbotAmazon3509.7%-102
YouBotYou.com3279.0%+2
ChatGPT-UserOpenAI3038.4%-7
DuckAssistBotDuckDuckGo2647.3%-3
Perplexity-UserPerplexity2607.2%-1
Claude-UserAnthropic2537.0%-1
MistralAI-UserMistral2506.9%-1
OAI-SearchBotOpenAI2446.7%-9
Claude-SearchBotAnthropic2436.7%-5
FirecrawlAgentFirecrawl2075.7%+3
TavilyBotTavily1915.3%-4
Google-NotebookLMGoogle1855.1%no change
kagi-fetcherKagi1584.4%-6

Policy changes this month

Where the index stood

The top and bottom of the leaderboard as it stands right now.

Most agent-ready

Least agent-ready

Citing this report

Free to quote and republish under CC BY 4.0. Credit Fidget Labs BV and include the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "The state of AI crawler access, September 2026." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.

Download the underlying dataset