Common Crawl . tier 2
CCBot
Feeds the Common Crawl corpus, which most open models train on.
585 of the 3,676 sites we measured deny CCBot access to their root, which is 15.9% of the index.
How to allow or block it
A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.
# Allow CCBot everywhere
User-agent: CCBot
Disallow:
# Block CCBot entirely
User-agent: CCBot
Disallow: /Sites blocking CCBot
Most-visited first. Showing up to 100 of 585.
| Rank | Domain | Score | Answer-surface crawlers | Stack |
|---|---|---|---|---|
| 5 | facebook.com | Score 38 out of 100, grade F, partial assessment | 6 blocked | unidentified |
| 11 | instagram.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 12 | mail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 16 | twitter.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 18 | linkedin.com | Score 18 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 25 | amazon.com | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 40 | netflix.com | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 52 | x.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 53 | pinterest.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 56 | tiktok.com | Score 15 out of 100, grade F | 11 blocked | unidentified |
| 60 | yahoo.comrefused GPTBot | Score 40 out of 100, grade D | 5 blocked | unidentified |
| 72 | chatgpt.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 74 | myfritz.net | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 80 | vimeo.com | Score 57 out of 100, grade D | 3 blocked | Next.js . Cloudflare |
| 86 | baidu.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 102 | reddit.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 119 | snapchat.com | Score 35 out of 100, grade F | 5 blocked | Next.js . Google Cloud / GFE |
| 120 | amazonvideo.com | Score 39 out of 100, grade F | 5 blocked | Amazon CloudFront |
| 147 | gravatar.com | Score 60 out of 100, grade C | 2 blocked | WordPress |
| 154 | nytimes.comrefused GPTBot | Score 38 out of 100, grade F | 11 blocked | WordPress . Fastly |
| 155 | tumblr.com | Score 43 out of 100, grade D | 4 blocked | WordPress |
| 161 | netflix.net | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 176 | flickr.com | Score 47 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 177 | soundcloud.com | Score 47 out of 100, grade D, partial assessment | 8 blocked | Amazon CloudFront |
| 188 | cnn.com | Score 32 out of 100, grade F | 10 blocked | Fastly |
| 189 | theguardian.com | Score 44 out of 100, grade D | 6 blocked | unidentified |
| 199 | ebay.com | Score 48 out of 100, grade D, partial assessment | 5 blocked | Akamai |
| 200 | forbes.com | Score 63 out of 100, grade C, partial assessment | 5 blocked | Next.js . Fastly |
| 207 | bbc.com | Score 38 out of 100, grade F | 9 blocked | Next.js . Fastly |
| 209 | t-online.de | Score 56 out of 100, grade D | 6 blocked | Next.js . Amazon CloudFront |
| 227 | bbc.co.uk | Score 41 out of 100, grade D | 9 blocked | Fastly |
| 237 | canva.comrefused GPTBot | Score 72 out of 100, grade C, partial assessment | 4 blocked | Cloudflare |
| 245 | weather.com | Score 71 out of 100, grade C | 6 blocked | WordPress . Akamai |
| 251 | who.int | Score 82 out of 100, grade B | all allowed | Cloudflare |
| 252 | imdb.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 264 | launchpad.netrefused GPTBot | Score 19 out of 100, grade F | 9 blocked | unidentified |
| 277 | reuters.com | Score 29 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 278 | forter.com | Score 90 out of 100, grade A | 1 blocked | HubSpot CMS . Cloudflare |
| 286 | nature.com | Score 34 out of 100, grade F | 7 blocked | Fastly |
| 292 | weibo.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 317 | amazon.co.uk | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 344 | naver.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 347 | trustpilot.com | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 348 | wsj.com | Score 35 out of 100, grade F, partial assessment | 8 blocked | unidentified |
| 356 | washingtonpost.com | Score 64 out of 100, grade C, partial assessment | 4 blocked | Next.js . Akamai |
| 359 | bloomberg.comrefused GPTBot | Score 33 out of 100, grade F | 10 blocked | Next.js . Fastly |
| 368 | vkontakte.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 373 | calendly.com | Score 87 out of 100, grade B | all allowed | Contentful . Cloudflare |
| 382 | amazon.de | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 393 | cookielaw.orgrefused GPTBot | Score 55 out of 100, grade D | 5 blocked | Adobe Experience Manager . Cloudflare |
| 416 | globo.com | Score 58 out of 100, grade D | 2 blocked | unidentified |
| 430 | nflxvideo.net | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 431 | businessinsider.com | Score 70 out of 100, grade C | 1 blocked | Fastly |
| 438 | espn.com | Score 47 out of 100, grade D | 3 blocked | Amazon CloudFront |
| 453 | cnbc.comrefused GPTBot | Score 30 out of 100, grade F | 9 blocked | Akamai |
| 455 | sohu.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 463 | goodreads.com | Score 58 out of 100, grade D | 1 blocked | Amazon CloudFront |
| 464 | gcdn.co | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 471 | edgecdn.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 474 | aol.comrefused GPTBot | Score 40 out of 100, grade D | 5 blocked | Next.js |
| 476 | quora.com | Score 14 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 481 | gosuslugi.ru | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 487 | scribd.com | Score 74 out of 100, grade C | 1 blocked | Next.js . Fastly |
| 497 | behance.net | Score 66 out of 100, grade C, partial assessment | 5 blocked | Fastly |
| 501 | amazon.co.jp | Score 15 out of 100, grade F | 10 blocked | Akamai |
| 507 | ft.comrefused GPTBot | Score 58 out of 100, grade D, partial assessment | 6 blocked | Cloudflare |
| 512 | slideshare.net | Score 52 out of 100, grade D | 1 blocked | Fastly |
| 525 | wired.com | Score 56 out of 100, grade D | 8 blocked | Cloudflare |
| 531 | usatoday.com | Score 24 out of 100, grade F | 11 blocked | Fastly |
| 537 | patreon.comrefused GPTBot | Score 39 out of 100, grade F | 6 blocked | Next.js . Cloudflare |
| 554 | tripadvisor.comrefused GPTBot | Score 55 out of 100, grade D | 4 blocked | Amazon CloudFront |
| 557 | uol.com.br | Score 52 out of 100, grade D | 8 blocked | Akamai |
| 567 | telegraph.co.uk | Score 30 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 568 | imgsmail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 601 | threads.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 602 | dailymail.co.ukrefused GPTBot | Score 30 out of 100, grade F | 11 blocked | Akamai |
| 612 | amazon.fr | Score 15 out of 100, grade F | 10 blocked | Akamai |
| 622 | nintendo.com | Score 48 out of 100, grade D | 5 blocked | Fastly |
| 626 | techcrunch.com | Score 64 out of 100, grade C | 5 blocked | WordPress |
| 629 | sagepub.com | Score 69 out of 100, grade C | 1 blocked | unidentified |
| 630 | sharethrough.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 641 | yelp.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 642 | amazon.ca | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 645 | elpais.comrefused GPTBot | Score 46 out of 100, grade D | 5 blocked | unidentified |
| 657 | cnet.com | Score 45 out of 100, grade D | 10 blocked | WordPress |
| 666 | amazon.in | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 673 | fb.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 679 | lemonde.fr | Score 53 out of 100, grade D | 5 blocked | Fastly |
| 681 | pinimg.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 683 | hihonorcloud.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 684 | quantserve.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 693 | dailymail.comrefused GPTBot | Score 30 out of 100, grade F | 11 blocked | Akamai |
| 707 | amazon.co.za | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 710 | creativecdn.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 712 | bandcamp.com | Score 48 out of 100, grade D | 4 blocked | Fastly |
| 714 | ted.com | Score 47 out of 100, grade D | 5 blocked | Next.js . Fastly |
| 724 | globalsign.com | Score 49 out of 100, grade D | 5 blocked | Cloudflare |
| 728 | amazon.it | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 734 | amazon.es | Score 15 out of 100, grade F | 10 blocked | Akamai |
| 749 | primevideo.com | Score 39 out of 100, grade F | 5 blocked | Amazon CloudFront |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Sites blocking CCBot." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.