Cross-tab
AI crawler blocking by top-level domain
A rough proxy for jurisdiction and market. It is not a clean one, since anyone anywhere can register a .com, but the differences between country TLDs are large enough to be worth looking at.
The spread is wide. fr sites block an answer-surface crawler 56.3% of the time. gov sites, 4.5%.
| Top-level domain | Sites | Blocking AI | Proportion blocking | Mean score |
|---|---|---|---|---|
| .com | 1,948 | 374 (19.2%) | 66.4 | |
| .org | 215 | 23 (10.7%) | 66.4 | |
| .net | 182 | 37 (20.3%) | 60.4 | |
| .ru | 145 | 13 (9.0%) | 61.5 | |
| .edu | 81 | 5 (6.2%) | 66.4 | |
| .io | 74 | 6 (8.1%) | 68 | |
| .de | 71 | 21 (29.6%) | 61.3 | |
| .gov | 66 | 3 (4.5%) | 66.8 | |
| .jp | 34 | 5 (14.7%) | 62.2 | |
| .fr | 32 | 18 (56.3%) | 56.8 | |
| .com.br | 30 | 4 (13.3%) | 68.4 | |
| .tv | 27 | 3 (11.1%) | 57.3 | |
| .it | 26 | 10 (38.5%) | 62.1 | |
| .co.uk | 25 | 14 (56.0%) | 52.4 | |
| .cn | 21 | 1 (4.8%) | 53 | |
| .me | 19 | 1 (5.3%) | 64.9 |
TLD is not nationality. Treat it as a weak signal about where an operator sits, not a strong one. Cohorts smaller than 25 measured sites are not shown: a blocking rate over a handful of domains is noise presented as a finding.
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "AI crawler blocking by top-level domain." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.