crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Report . August 2026

The state of AI crawler access, August 2026

Drawn from 21 daily crawls of up to 3,707 Tranco-ranked domains, of which 3,652 could be measured.

Sealed record. This month closed and was written to a permanent file on , under rubric 2.0.0 and probe 3.0.0. Nothing on this page will change again, including the cross-tabs and the change list. Cite it freely. The sealed JSON.

Research and data by Fidget Labs BV, Breda, Netherlands. Published as CrawlIndex and free to reuse under CC BY 4.0 with attribution.

18.3%
Block at least one answer-surface crawler
669 sites
4.1%
Block every answer-surface crawler
151 sites
12.8%
Publish an llms.txt
469 sites
63
Mean readiness score
Out of 100

What moved

Between 2026-08-10 and 2026-08-31, the number of measured sites blocking at least one answer-surface AI crawler rose by 21 sites. Sites publishing an llms.txt rose by 24 sites, and sites publishing an agents.md rose by 8 sites.

The mean readiness score across scored sites fell 5.41 points to 63.47.

10 sites answered an agent with HTTP 402 Payment Required. That is a pay-per-crawl gateway rather than a block: access is being metered and sold rather than refused.

Sites that could not be measured are excluded rather than counted as failures, so these totals are over the 3,652 domains actually observed. Method: /methodology. Coverage and its limits: /coverage.

Across the month

Blocking an answer-surface crawler648 to 669 over 21 crawls
Publishing llms.txt445 to 469 over 21 crawls
Mean readiness score68.88 to 63.47 over 21 crawls

Who set the policy

Blocking rate grouped by edge network. Sites behind varnish blocked at 25.6% against 3.2% behind alibaba, which is a far wider spread than anything explained by what the sites themselves publish.

Blocking rate by edge network
Edge networkSitesBlocking AIProportion blockingMean score
Cloudflare1,120206 (18.4%)68.8
Amazon CloudFront41986 (20.5%)64.3
Akamai26341 (15.6%)67.3
Fastly21053 (25.2%)63.7
Google Cloud / GFE1506 (4.0%)57.8
Vercel492 (4.1%)73.9
Varnish3910 (25.6%)60.7
Alibaba Cloud CDN311 (3.2%)63.2
Azure Front Door281 (3.6%)60.9
Netlify251 (4.0%)70.6

By publishing platform

Blocking rate by publishing platform
PlatformSitesBlocking AIProportion blockingMean score
WordPress30050 (16.7%)69.7
Next.js28648 (16.8%)65.9
Adobe Experience Manager1327 (5.3%)65.7
Drupal1156 (5.2%)63.2
HubSpot CMS816 (7.4%)71.2
Nuxt544 (7.4%)62.7
Contentful512 (3.9%)67
Sanity282 (7.1%)74.3

Blocking by crawler

AI crawlers by number of sites blocking them in August 2026
CrawlerOperatorBlocked byShareOver period
CCBotCommon Crawl59416.3%+23
GPTBotOpenAI56415.4%+20
BytespiderByteDance56015.3%+26
ClaudeBotAnthropic53714.7%+19
meta-externalagentMeta50513.8%+24
Google-ExtendedGoogle49213.5%+20
Applebot-ExtendedApple46912.8%+24
AmazonbotAmazon45312.4%+21
DiffbotDiffbot37810.4%+3
cohere-aiCohere37110.2%-2
PerplexityBotPerplexity3639.9%-10
YouBotYou.com3258.9%+2
ChatGPT-UserOpenAI3118.5%-10
DuckAssistBotDuckDuckGo2687.3%-4
Perplexity-UserPerplexity2617.1%-8
Claude-UserAnthropic2557.0%-8
MistralAI-UserMistral2536.9%-2
OAI-SearchBotOpenAI2526.9%-4
Claude-SearchBotAnthropic2506.8%-7
FirecrawlAgentFirecrawl2045.6%no change
TavilyBotTavily1955.3%no change
Google-NotebookLMGoogle1855.1%-1
kagi-fetcherKagi1644.5%-1

Policy changes this month

Where the index stood

The top and bottom of the leaderboard at the moment this month was sealed.

Most agent-ready

Least agent-ready

Citing this report

Free to quote and republish under CC BY 4.0. Credit Fidget Labs BV and include the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "The state of AI crawler access, August 2026." https://crawlindex.org (measured 2026-08-31). Licensed CC BY 4.0.

Download the underlying dataset