Report . August 2026
The state of AI crawler access, August 2026
Drawn from 1 daily crawl of up to 4,984 Tranco-ranked domains, of which 3,633 could be measured.
Research and data by Fidget Labs BV, Breda, Netherlands. Published as CrawlIndex and free to reuse under CC BY 4.0 with attribution.
What moved
This report covers a single day of measurement, so it records a starting position rather than a trend. Month-on-month movement appears here once the index has more than one crawl inside the period.
11 sites answered an agent with HTTP 402 Payment Required. That is a pay-per-crawl gateway rather than a block: access is being metered and sold rather than refused.
Sites that could not be measured are excluded rather than counted as failures, so these totals are over the 3,633 domains actually observed. Method: /methodology.
Who set the policy
Blocking rate grouped by edge network. Sites behind varnish blocked at 30.8% against 3.3% behind google, which is a far wider spread than anything explained by what the sites themselves publish.
| Edge network | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| Cloudflare | 1,082 | 187 (17.3%) | 72 | |
| Amazon CloudFront | 413 | 82 (19.9%) | 64.2 | |
| Akamai | 265 | 43 (16.2%) | 69.7 | |
| Fastly | 216 | 53 (24.5%) | 66.3 | |
| Google Cloud / GFE | 153 | 5 (3.3%) | 57.8 | |
| Vercel | 48 | 3 (6.3%) | 77.2 | |
| Varnish | 39 | 12 (30.8%) | 60.4 | |
| Alibaba Cloud CDN | 37 | 5 (13.5%) | 52.7 | |
| Azure Front Door | 28 | 1 (3.6%) | 62.4 | |
| Netlify | 25 | 1 (4.0%) | 73.7 |
By publishing platform
| Platform | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| WordPress | 300 | 52 (17.3%) | 72.3 | |
| Next.js | 280 | 43 (15.4%) | 68.4 | |
| Adobe Experience Manager | 130 | 6 (4.6%) | 68.6 | |
| Drupal | 114 | 2 (1.8%) | 66.1 | |
| HubSpot CMS | 80 | 3 (3.8%) | 74.6 | |
| Nuxt | 57 | 4 (7.0%) | 64.3 | |
| Contentful | 50 | 1 (2.0%) | 70.6 | |
| Sanity | 30 | 3 (10.0%) | 78.7 |
Blocking by crawler
| Crawler | Operator | Blocked by | Share | Over period |
|---|---|---|---|---|
| CCBot | Common Crawl | 574 | 15.8% | no change |
| GPTBot | OpenAI | 547 | 15.1% | no change |
| Bytespider | ByteDance | 538 | 14.8% | no change |
| ClaudeBot | Anthropic | 522 | 14.4% | no change |
| meta-externalagent | Meta | 485 | 13.3% | no change |
| Google-Extended | 473 | 13.0% | no change | |
| Applebot-Extended | Apple | 447 | 12.3% | no change |
| Amazonbot | Amazon | 439 | 12.1% | no change |
| cohere-ai | Cohere | 378 | 10.4% | no change |
| PerplexityBot | Perplexity | 376 | 10.3% | no change |
| Diffbot | Diffbot | 375 | 10.3% | no change |
| YouBot | You.com | 327 | 9.0% | no change |
| ChatGPT-User | OpenAI | 321 | 8.8% | no change |
| DuckAssistBot | DuckDuckGo | 274 | 7.5% | no change |
| Perplexity-User | Perplexity | 273 | 7.5% | no change |
| Claude-User | Anthropic | 266 | 7.3% | no change |
| OAI-SearchBot | OpenAI | 260 | 7.2% | no change |
| Claude-SearchBot | Anthropic | 260 | 7.2% | no change |
| MistralAI-User | Mistral | 258 | 7.1% | no change |
| FirecrawlAgent | Firecrawl | 206 | 5.7% | no change |
| TavilyBot | Tavily | 197 | 5.4% | no change |
| Google-NotebookLM | 188 | 5.2% | no change | |
| kagi-fetcher | Kagi | 167 | 4.6% | no change |
Citing this report
Free to quote and republish under CC BY 4.0. Credit Fidget Labs BV and include the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "The state of AI crawler access, August 2026." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.