Report . October 2026
The state of AI crawler access, October 2026
Drawn from 31 daily crawls of up to 3,713 Tranco-ranked domains, of which 3,686 could be measured.
Sealed record. This month closed and was written to a permanent file on , under rubric 2.0.0 and probe 3.0.0. Nothing on this page will change again, including the cross-tabs and the change list. Cite it freely. The sealed JSON.
Research and data by Fidget Labs BV, Breda, Netherlands. Published as CrawlIndex and free to reuse under CC BY 4.0 with attribution.
What moved
Between 2026-10-01 and 2026-10-31, the number of measured sites blocking at least one answer-surface AI crawler rose by 74 sites. Sites publishing an llms.txt rose by 32 sites, and sites publishing an agents.md fell by 3 sites.
The mean readiness score across scored sites fell 0.07 points to 64.21.
9 sites answered an agent with HTTP 402 Payment Required. That is a pay-per-crawl gateway rather than a block: access is being metered and sold rather than refused.
Sites that could not be measured are excluded rather than counted as failures, so these totals are over the 3,686 domains actually observed. Method: /methodology. Coverage and its limits: /coverage.
Across the month
Who set the policy
Blocking rate grouped by edge network. Sites behind varnish blocked at 26.5% against 2.5% behind google, which is a far wider spread than anything explained by what the sites themselves publish.
| Edge network | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| Cloudflare | 1,139 | 181 (15.9%) | 69.8 | |
| Amazon CloudFront | 427 | 92 (21.5%) | 64.5 | |
| Akamai | 280 | 40 (14.3%) | 68.4 | |
| Fastly | 217 | 53 (24.4%) | 64.1 | |
| Google Cloud / GFE | 161 | 4 (2.5%) | 60.1 | |
| Vercel | 56 | 2 (3.6%) | 75.4 | |
| Varnish | 34 | 9 (26.5%) | 62.4 | |
| Azure Front Door | 32 | 2 (6.3%) | 60 | |
| Alibaba Cloud CDN | 31 | 2 (6.5%) | 65.5 |
By publishing platform
| Platform | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| Next.js | 311 | 55 (17.7%) | 66.2 | |
| WordPress | 306 | 50 (16.3%) | 70.1 | |
| Adobe Experience Manager | 137 | 8 (5.8%) | 65.1 | |
| Drupal | 110 | 5 (4.5%) | 64.1 | |
| HubSpot CMS | 84 | 4 (4.8%) | 72.9 | |
| Contentful | 56 | 3 (5.4%) | 67.9 | |
| Nuxt | 50 | 1 (2.0%) | 64.5 | |
| Sanity | 31 | 3 (9.7%) | 76.7 |
Blocking by crawler
| Crawler | Operator | Blocked by | Share | Over period |
|---|---|---|---|---|
| CCBot | Common Crawl | 570 | 15.5% | +78 |
| Bytespider | ByteDance | 551 | 14.9% | +89 |
| GPTBot | OpenAI | 537 | 14.6% | +70 |
| ClaudeBot | Anthropic | 516 | 14.0% | +79 |
| meta-externalagent | Meta | 488 | 13.2% | +88 |
| Google-Extended | 467 | 12.7% | +81 | |
| Diffbot | Diffbot | 466 | 12.6% | +92 |
| Applebot-Extended | Apple | 452 | 12.3% | +88 |
| cohere-ai | Cohere | 451 | 12.2% | +81 |
| Amazonbot | Amazon | 440 | 11.9% | +91 |
| PerplexityBot | Perplexity | 361 | 9.8% | +14 |
| YouBot | You.com | 334 | 9.1% | +11 |
| ChatGPT-User | OpenAI | 305 | 8.3% | +7 |
| Perplexity-User | Perplexity | 273 | 7.4% | +20 |
| Claude-User | Anthropic | 270 | 7.3% | +22 |
| DuckAssistBot | DuckDuckGo | 269 | 7.3% | +10 |
| MistralAI-User | Mistral | 260 | 7.1% | +16 |
| Claude-SearchBot | Anthropic | 257 | 7.0% | +14 |
| OAI-SearchBot | OpenAI | 254 | 6.9% | +11 |
| FirecrawlAgent | Firecrawl | 216 | 5.9% | +12 |
| TavilyBot | Tavily | 194 | 5.3% | +4 |
| Google-NotebookLM | 190 | 5.2% | +9 | |
| kagi-fetcher | Kagi | 161 | 4.4% | +2 |
Policy changes this month
- zscaler.comzscaler.com: llms.txt added.
- zebra.comzebra.com: llms.txt added.
- yieldmo.comyieldmo.com: llms.txt removed.
- umd.eduumd.edu now allows Google-Extended.
- trip.comtrip.com: llms.txt removed.
- themegrill.comthemegrill.com: llms.txt removed.
- tappx.comtappx.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- hihonor.comhihonor.com: llms.txt added.
- farfetch.comfarfetch.com: llms.txt added.
- eltiempo.eseltiempo.es: llms.txt removed.
- comix.tocomix.to now allows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- coindesk.comcoindesk.com now blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent.
- brandmetrics.combrandmetrics.com: llms.txt removed.
- biorxiv.orgbiorxiv.org now allows GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot.
- adroll.comadroll.com: llms.txt removed.
- zebra.comzebra.com: llms.txt removed.
- tranche.financetranche.finance now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- theregister.comtheregister.com now blocks GPTBot, ClaudeBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- ssrn.comssrn.com now blocks GPTBot, ChatGPT-User, Google-Extended.
- producthunt.comproducthunt.com: llms.txt added.
- prezi.comprezi.com now blocks Applebot-Extended, meta-externalagent.
- pnas.orgpnas.org now allows GPTBot.
- newsweek.comnewsweek.com now blocks Perplexity-User.
- meraki.commeraki.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- marca.commarca.com now blocks ClaudeBot, Claude-User, Claude-SearchBot, Perplexity-User, Applebot-Extended, meta-externalagent.
- hihonor.comhihonor.com: llms.txt removed.
- gosuslugi.rugosuslugi.ru now blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- gamespot.comgamespot.com now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- farfetch.comfarfetch.com: llms.txt removed.
- express.co.ukexpress.co.uk now blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot-Extended.
- eff.orgeff.org: llms.txt added.
- dw.comdw.com now blocks Perplexity-User.
- codepen.iocodepen.io now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- bloomberg.combloomberg.com: llms.txt removed.
- blender.orgblender.org now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- blacktoon423.comblacktoon423.com now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- bizjournals.combizjournals.com now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent.
- biorxiv.orgbiorxiv.org now blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot.
- arlo.comarlo.com: llms.txt added.
- amazonvideo.comamazonvideo.com now blocks ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, meta-externalagent.
Where the index stood
The top and bottom of the leaderboard at the moment this month was sealed.
Most agent-ready
- Score 95 out of 100, grade Acrawlindex.org
- Score 93 out of 100, grade Atherapservices.net
- Score 93 out of 100, grade Ayotpo.com
- Score 93 out of 100, grade Aheroku.com
- Score 92 out of 100, grade Arailway.app
- Score 91 out of 100, grade Aatlassian.com
- Score 91 out of 100, grade Ausercentrics.eu
- Score 91 out of 100, grade Ahaberler.com
- Score 91 out of 100, grade Aprotothema.gr
- Score 91 out of 100, grade Asondakika.com
Least agent-ready
- Score 15 out of 100, grade Ftiktok.com
- Score 15 out of 100, grade Famazon.com.au
- Score 19 out of 100, grade Ffindagrave.com
- Score 20 out of 100, grade Flaunchpad.net
- Score 21 out of 100, grade Fgentoo.org
- Score 22 out of 100, grade Famazon.fr
- Score 22 out of 100, grade Famazon.co.za
- Score 22 out of 100, grade Famazon.eg
- Score 22 out of 100, grade Famazon.ae
- Score 22 out of 100, grade Famazon.sa
Citing this report
Free to quote and republish under CC BY 4.0. Credit Fidget Labs BV and include the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "The state of AI crawler access, October 2026." https://crawlindex.org (measured 2026-10-31). Licensed CC BY 4.0.