crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

OpenAI . tier 1

GPTBot

Collects training data for OpenAI models.

533 of the 3,656 sites we measured deny GPTBot access to their root, which is 14.6% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow GPTBot everywhere
User-agent: GPTBot
Disallow:

# Block GPTBot entirely
User-agent: GPTBot
Disallow: /

Sites blocking GPTBot

Most-visited first. Showing up to 100 of 533.

Domains that block GPTBot
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
29amazon.comrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
63tiktok.comScore 15 out of 100, grade F11 blockedunidentified
72yahoo.comrefused GPTBotScore 43 out of 100, grade D5 blockedNext.js
84vimeo.comScore 56 out of 100, grade D3 blockedNext.js . Cloudflare
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
96t-online.deScore 55 out of 100, grade D5 blockedNext.js . Amazon CloudFront
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
155nytimes.comrefused GPTBotScore 21 out of 100, grade F, partial assessment11 blockedFastly
159snapchat.comScore 46 out of 100, grade D5 blockedNext.js . Google Cloud / GFE
167gravatar.comScore 60 out of 100, grade C2 blockedWordPress
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
192cnn.comScore 31 out of 100, grade F9 blockedFastly
198ebay.comScore 60 out of 100, grade C, partial assessment5 blockedAkamai
202soundcloud.comScore 31 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
209medium.comrefused GPTBotScore 46 out of 100, grade D4 blockedCloudflare
219bbc.comScore 31 out of 100, grade F10 blockedNext.js . Fastly
227forbes.comrefused GPTBotScore 57 out of 100, grade D, partial assessment5 blockedFastly
242bbc.co.ukScore 34 out of 100, grade F10 blockedFastly
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
294nature.comScore 45 out of 100, grade D, partial assessment7 blockedFastly
296launchpad.netrefused GPTBotScore 20 out of 100, grade F9 blockedunidentified
298canva.comrefused GPTBotScore 66 out of 100, grade C, partial assessment4 blockedCloudflare
327alibaba.comScore 53 out of 100, grade D2 blockedAlibaba Cloud CDN
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
333amazon.co.ukrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
340espn.comScore 72 out of 100, grade C, partial assessment3 blockedAmazon CloudFront
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
386amazon.derefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
392globo.comScore 56 out of 100, grade D2 blockedunidentified
393go.comScore 56 out of 100, grade D2 blockedAmazon CloudFront
404bloomberg.comrefused GPTBotScore 27 out of 100, grade F, partial assessment10 blockedFastly
413claude.airefused GPTBotScore 51 out of 100, grade D4 blockedCloudflare
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
430aliexpress.comScore 71 out of 100, grade C, partial assessment3 blockedAlibaba Cloud CDN
433aol.comScore 50 out of 100, grade D3 blockedNext.js . Amazon CloudFront
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
462yahoo.co.jpScore 55 out of 100, grade D2 blockedunidentified
467pixabay.comrefused GPTBotScore 67 out of 100, grade C, partial assessment4 blockedCloudflare
479goodreads.comScore 56 out of 100, grade D1 blockedAmazon CloudFront
486cnbc.comrefused GPTBotScore 24 out of 100, grade F11 blockedAkamai
495amazon.co.jprefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
504quora.comScore 13 out of 100, grade F, partial assessment10 blockedunidentified
509statista.comrefused GPTBotScore 43 out of 100, grade D7 blockedAmazon CloudFront
534tandfonline.comrefused GPTBotScore 86 out of 100, grade B, partial assessment1 blockedCloudflare
536behance.netrefused GPTBotScore 60 out of 100, grade C, partial assessment5 blockedFastly
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
569sohu.comScore 8 out of 100, grade F, partial assessment11 blockedunidentified
582dailymail.comrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
601tripadvisor.comrefused GPTBotScore 63 out of 100, grade C, partial assessment4 blockedAmazon CloudFront
622techcrunch.comScore 66 out of 100, grade C3 blockedWordPress . Cloudflare
629usatoday.comScore 16 out of 100, grade F, partial assessment11 blockedunidentified
630telegraph.co.ukScore 37 out of 100, grade F10 blockedAdobe Experience Manager . Fastly
632amazon.frrefused GPTBotScore 15 out of 100, grade F10 blockedAmazon CloudFront
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
644amazon.carefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
668nintendo.comScore 48 out of 100, grade D5 blockedNext.js . Fastly
675amazon.inrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
690spiegel.deScore 40 out of 100, grade D8 blockedGoogle Cloud / GFE
691cbsnews.comScore 74 out of 100, grade C1 blockedVarnish
692kick.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
700bandcamp.comScore 46 out of 100, grade D4 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
707globalsign.comScore 56 out of 100, grade D5 blockedFastly
708hugedomains.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
720amazon.esrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
730cookielaw.orgrefused GPTBotScore 51 out of 100, grade D5 blockedAdobe Experience Manager . Cloudflare
734dailymail.co.ukrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
736amazon.itrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
743wattpad.comScore 58 out of 100, grade D1 blockedRemix / React Router . Amazon CloudFront
756cnet.comScore 44 out of 100, grade D10 blockedWordPress
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
770apnews.comrefused GPTBotScore 49 out of 100, grade D, partial assessment7 blockedCloudflare
798wp.plScore 57 out of 100, grade D1 blockedunidentified
800sharethrough.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
804airbnb.comScore 56 out of 100, grade D3 blockedunidentified
811attn.tvScore 6 out of 100, grade F, partial assessment11 blockedunidentified
838ted.comScore 52 out of 100, grade D5 blockedNext.js . Fastly
840ebay.deScore 60 out of 100, grade C, partial assessment5 blockedAkamai
845salesforceliveagent.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
849adform.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
856academia.eduScore 48 out of 100, grade D, partial assessment6 blockedunidentified
868imgsmail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
878pexels.comrefused GPTBotScore 72 out of 100, grade C, partial assessment3 blockedCloudflare
882nbcnews.comScore 30 out of 100, grade F11 blockedNext.js . Akamai

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking GPTBot." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.