Cross-tab
Does your CDN decide your AI policy?
Most operators never sat down and formed a view on AI crawlers. Their edge network shipped a default and they inherited it. Grouped from response headers, which are far harder to misread than markup.
The spread is wide. fastly sites block an answer-surface crawler 24.8% of the time. google sites, 2.5%.
| Edge network | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| Cloudflare | 1,130 | 180 (15.9%) | 69.7 | |
| Amazon CloudFront | 421 | 82 (19.5%) | 64.8 | |
| Akamai | 278 | 42 (15.1%) | 67.4 | |
| Fastly | 214 | 53 (24.8%) | 63.9 | |
| Google Cloud / GFE | 163 | 4 (2.5%) | 58 | |
| Vercel | 58 | 2 (3.4%) | 75.5 | |
| Varnish | 33 | 8 (24.2%) | 62.9 | |
| Alibaba Cloud CDN | 32 | 2 (6.3%) | 64.8 | |
| Azure Front Door | 31 | 2 (6.5%) | 60.1 |
Networks are identified from response headers only. Cohorts smaller than 25 measured sites are not shown: a blocking rate over a handful of domains is noise presented as a finding.
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Does your CDN decide your AI policy?." https://crawlindex.org (measured 2026-11-07). Licensed CC BY 4.0.