OpenAI . tier 1
GPTBot
Collects training data for OpenAI models.
533 of the 3,656 sites we measured deny GPTBot access to their root, which is 14.6% of the index.
How to allow or block it
A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.
# Allow GPTBot everywhere
User-agent: GPTBot
Disallow:
# Block GPTBot entirely
User-agent: GPTBot
Disallow: /Sites blocking GPTBot
Most-visited first. Showing up to 100 of 533.
| Rank | Domain | Score | Answer-surface crawlers | Stack |
|---|---|---|---|---|
| 5 | facebook.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 14 | instagram.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 16 | twitter.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 17 | linkedin.com | Score 17 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 27 | mail.ru | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 29 | amazon.comrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 48 | x.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 55 | pinterest.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 62 | myfritz.net | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 63 | tiktok.com | Score 15 out of 100, grade F | 11 blocked | unidentified |
| 72 | yahoo.comrefused GPTBot | Score 43 out of 100, grade D | 5 blocked | Next.js |
| 84 | vimeo.com | Score 56 out of 100, grade D | 3 blocked | Next.js . Cloudflare |
| 90 | baidu.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 96 | t-online.de | Score 55 out of 100, grade D | 5 blocked | Next.js . Amazon CloudFront |
| 111 | reddit.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 114 | chatgpt.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 155 | nytimes.comrefused GPTBot | Score 21 out of 100, grade F, partial assessment | 11 blocked | Fastly |
| 159 | snapchat.com | Score 46 out of 100, grade D | 5 blocked | Next.js . Google Cloud / GFE |
| 167 | gravatar.com | Score 60 out of 100, grade C | 2 blocked | WordPress |
| 170 | skybridge.click | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 181 | flickr.com | Score 42 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 192 | cnn.com | Score 31 out of 100, grade F | 9 blocked | Fastly |
| 198 | ebay.com | Score 60 out of 100, grade C, partial assessment | 5 blocked | Akamai |
| 202 | soundcloud.com | Score 31 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 209 | medium.comrefused GPTBot | Score 46 out of 100, grade D | 4 blocked | Cloudflare |
| 219 | bbc.com | Score 31 out of 100, grade F | 10 blocked | Next.js . Fastly |
| 227 | forbes.comrefused GPTBot | Score 57 out of 100, grade D, partial assessment | 5 blocked | Fastly |
| 242 | bbc.co.uk | Score 34 out of 100, grade F | 10 blocked | Fastly |
| 265 | imdb.com | Score 9 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 285 | reuters.com | Score 26 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 294 | nature.com | Score 45 out of 100, grade D, partial assessment | 7 blocked | Fastly |
| 296 | launchpad.netrefused GPTBot | Score 20 out of 100, grade F | 9 blocked | unidentified |
| 298 | canva.comrefused GPTBot | Score 66 out of 100, grade C, partial assessment | 4 blocked | Cloudflare |
| 327 | alibaba.com | Score 53 out of 100, grade D | 2 blocked | Alibaba Cloud CDN |
| 331 | weibo.com | Score 18 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 333 | amazon.co.ukrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 340 | espn.com | Score 72 out of 100, grade C, partial assessment | 3 blocked | Amazon CloudFront |
| 351 | naver.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 386 | amazon.derefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 392 | globo.com | Score 56 out of 100, grade D | 2 blocked | unidentified |
| 393 | go.com | Score 56 out of 100, grade D | 2 blocked | Amazon CloudFront |
| 404 | bloomberg.comrefused GPTBot | Score 27 out of 100, grade F, partial assessment | 10 blocked | Fastly |
| 413 | claude.airefused GPTBot | Score 51 out of 100, grade D | 4 blocked | Cloudflare |
| 425 | gcdn.co | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 430 | aliexpress.com | Score 71 out of 100, grade C, partial assessment | 3 blocked | Alibaba Cloud CDN |
| 433 | aol.com | Score 50 out of 100, grade D | 3 blocked | Next.js . Amazon CloudFront |
| 458 | netcraze.io | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 462 | yahoo.co.jp | Score 55 out of 100, grade D | 2 blocked | unidentified |
| 467 | pixabay.comrefused GPTBot | Score 67 out of 100, grade C, partial assessment | 4 blocked | Cloudflare |
| 479 | goodreads.com | Score 56 out of 100, grade D | 1 blocked | Amazon CloudFront |
| 486 | cnbc.comrefused GPTBot | Score 24 out of 100, grade F | 11 blocked | Akamai |
| 495 | amazon.co.jprefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 504 | quora.com | Score 13 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 509 | statista.comrefused GPTBot | Score 43 out of 100, grade D | 7 blocked | Amazon CloudFront |
| 534 | tandfonline.comrefused GPTBot | Score 86 out of 100, grade B, partial assessment | 1 blocked | Cloudflare |
| 536 | behance.netrefused GPTBot | Score 60 out of 100, grade C, partial assessment | 5 blocked | Fastly |
| 565 | threads.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 569 | sohu.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 582 | dailymail.comrefused GPTBot | Score 35 out of 100, grade F | 11 blocked | Akamai |
| 601 | tripadvisor.comrefused GPTBot | Score 63 out of 100, grade C, partial assessment | 4 blocked | Amazon CloudFront |
| 622 | techcrunch.com | Score 66 out of 100, grade C | 3 blocked | WordPress . Cloudflare |
| 629 | usatoday.com | Score 16 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 630 | telegraph.co.uk | Score 37 out of 100, grade F | 10 blocked | Adobe Experience Manager . Fastly |
| 632 | amazon.frrefused GPTBot | Score 15 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 636 | edgecdn.ru | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 644 | amazon.carefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 667 | hstgr.net | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 668 | nintendo.com | Score 48 out of 100, grade D | 5 blocked | Next.js . Fastly |
| 675 | amazon.inrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 690 | spiegel.de | Score 40 out of 100, grade D | 8 blocked | Google Cloud / GFE |
| 691 | cbsnews.com | Score 74 out of 100, grade C | 1 blocked | Varnish |
| 692 | kick.comrefused GPTBot | Score 61 out of 100, grade C, partial assessment | 5 blocked | Cloudflare |
| 700 | bandcamp.com | Score 46 out of 100, grade D | 4 blocked | Fastly |
| 702 | fb.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 706 | yelp.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 707 | globalsign.com | Score 56 out of 100, grade D | 5 blocked | Fastly |
| 708 | hugedomains.comrefused GPTBot | Score 61 out of 100, grade C, partial assessment | 5 blocked | Cloudflare |
| 720 | amazon.esrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 724 | quantserve.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 729 | pinimg.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 730 | cookielaw.orgrefused GPTBot | Score 51 out of 100, grade D | 5 blocked | Adobe Experience Manager . Cloudflare |
| 734 | dailymail.co.ukrefused GPTBot | Score 35 out of 100, grade F | 11 blocked | Akamai |
| 736 | amazon.itrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 741 | creativecdn.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 743 | wattpad.com | Score 58 out of 100, grade D | 1 blocked | Remix / React Router . Amazon CloudFront |
| 756 | cnet.com | Score 44 out of 100, grade D | 10 blocked | WordPress |
| 762 | hihonorcloud.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 770 | apnews.comrefused GPTBot | Score 49 out of 100, grade D, partial assessment | 7 blocked | Cloudflare |
| 798 | wp.pl | Score 57 out of 100, grade D | 1 blocked | unidentified |
| 800 | sharethrough.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 804 | airbnb.com | Score 56 out of 100, grade D | 3 blocked | unidentified |
| 811 | attn.tv | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 838 | ted.com | Score 52 out of 100, grade D | 5 blocked | Next.js . Fastly |
| 840 | ebay.de | Score 60 out of 100, grade C, partial assessment | 5 blocked | Akamai |
| 845 | salesforceliveagent.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 849 | adform.net | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 856 | academia.edu | Score 48 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 868 | imgsmail.ru | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 878 | pexels.comrefused GPTBot | Score 72 out of 100, grade C, partial assessment | 3 blocked | Cloudflare |
| 882 | nbcnews.com | Score 30 out of 100, grade F | 11 blocked | Next.js . Akamai |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Sites blocking GPTBot." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.