crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Apple . tier 1

Applebot-Extended

Controls use in Apple Intelligence training.

455 of the 3,656 sites we measured deny Applebot-Extended access to their root, which is 12.4% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow Applebot-Extended everywhere
User-agent: Applebot-Extended
Disallow:

# Block Applebot-Extended entirely
User-agent: Applebot-Extended
Disallow: /

Sites blocking Applebot-Extended

Most-visited first. Showing up to 100 of 455.

Domains that block Applebot-Extended
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
43netflix.comScore 59 out of 100, grade D, partial assessment3 blockedunidentified
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
63tiktok.comScore 15 out of 100, grade F11 blockedunidentified
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
96t-online.deScore 55 out of 100, grade D5 blockedNext.js . Amazon CloudFront
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
153tumblr.comScore 54 out of 100, grade D4 blockedWordPress
155nytimes.comrefused GPTBotScore 21 out of 100, grade F, partial assessment11 blockedFastly
159snapchat.comScore 46 out of 100, grade D5 blockedNext.js . Google Cloud / GFE
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
192cnn.comScore 31 out of 100, grade F9 blockedFastly
198ebay.comScore 60 out of 100, grade C, partial assessment5 blockedAkamai
202soundcloud.comScore 31 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
203theguardian.comScore 46 out of 100, grade D6 blockedunidentified
209medium.comrefused GPTBotScore 46 out of 100, grade D4 blockedCloudflare
219bbc.comScore 31 out of 100, grade F10 blockedNext.js . Fastly
227forbes.comrefused GPTBotScore 57 out of 100, grade D, partial assessment5 blockedFastly
239researchgate.netrefused GPTBotScore 78 out of 100, grade B, partial assessment1 blockedCloudflare
242bbc.co.ukScore 34 out of 100, grade F10 blockedFastly
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
294nature.comScore 45 out of 100, grade D, partial assessment7 blockedFastly
296launchpad.netrefused GPTBotScore 20 out of 100, grade F9 blockedunidentified
298canva.comrefused GPTBotScore 66 out of 100, grade C, partial assessment4 blockedCloudflare
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
332nflxvideo.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
341trustpilot.comScore 66 out of 100, grade C, partial assessment2 blockedunidentified
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
371wsj.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
404bloomberg.comrefused GPTBotScore 27 out of 100, grade F, partial assessment10 blockedFastly
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
467pixabay.comrefused GPTBotScore 67 out of 100, grade C, partial assessment4 blockedCloudflare
486cnbc.comrefused GPTBotScore 24 out of 100, grade F11 blockedAkamai
504quora.comScore 13 out of 100, grade F, partial assessment10 blockedunidentified
509statista.comrefused GPTBotScore 43 out of 100, grade D7 blockedAmazon CloudFront
536behance.netrefused GPTBotScore 60 out of 100, grade C, partial assessment5 blockedFastly
555ft.comrefused GPTBotScore 53 out of 100, grade D, partial assessment6 blockedCloudflare
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
569sohu.comScore 8 out of 100, grade F, partial assessment11 blockedunidentified
572wired.comScore 51 out of 100, grade D8 blockedCloudflare
582dailymail.comrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
600uol.com.brScore 47 out of 100, grade D8 blockedAmazon CloudFront
601tripadvisor.comrefused GPTBotScore 63 out of 100, grade C, partial assessment4 blockedAmazon CloudFront
622techcrunch.comScore 66 out of 100, grade C3 blockedWordPress . Cloudflare
629usatoday.comScore 16 out of 100, grade F, partial assessment11 blockedunidentified
630telegraph.co.ukScore 37 out of 100, grade F10 blockedAdobe Experience Manager . Fastly
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
668nintendo.comScore 48 out of 100, grade D5 blockedNext.js . Fastly
690spiegel.deScore 40 out of 100, grade D8 blockedGoogle Cloud / GFE
692kick.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
695lemonde.frScore 48 out of 100, grade D, partial assessment5 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
707globalsign.comScore 56 out of 100, grade D5 blockedFastly
708hugedomains.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
730cookielaw.orgrefused GPTBotScore 51 out of 100, grade D5 blockedAdobe Experience Manager . Cloudflare
734dailymail.co.ukrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
749netflix.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
756cnet.comScore 44 out of 100, grade D10 blockedWordPress
758bild.deScore 59 out of 100, grade D3 blockedunidentified
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
770apnews.comrefused GPTBotScore 49 out of 100, grade D, partial assessment7 blockedCloudflare
786welt.deScore 50 out of 100, grade D3 blockedunidentified
790zillow.comrefused GPTBotScore 50 out of 100, grade D2 blockedNext.js . Amazon CloudFront
800sharethrough.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
804airbnb.comScore 56 out of 100, grade D3 blockedunidentified
811attn.tvScore 6 out of 100, grade F, partial assessment11 blockedunidentified
825people.comScore 35 out of 100, grade F, partial assessment8 blockedCloudflare
838ted.comScore 52 out of 100, grade D5 blockedNext.js . Fastly
840ebay.deScore 60 out of 100, grade C, partial assessment5 blockedAkamai
845salesforceliveagent.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
849adform.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
853theverge.comScore 44 out of 100, grade D9 blockedWordPress . Fastly
856academia.eduScore 48 out of 100, grade D, partial assessment6 blockedunidentified
868imgsmail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
878pexels.comrefused GPTBotScore 72 out of 100, grade C, partial assessment3 blockedCloudflare
882nbcnews.comScore 30 out of 100, grade F11 blockedNext.js . Akamai
884ebay.co.ukScore 60 out of 100, grade C, partial assessment5 blockedAkamai
914hbr.orgrefused GPTBotScore 38 out of 100, grade F5 blockedNext.js . Amazon CloudFront
935daum.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
939repubblica.itScore 65 out of 100, grade C4 blockedAmazon CloudFront
954huffpost.comScore 46 out of 100, grade D11 blockedFastly
955corriere.itrefused GPTBotScore 34 out of 100, grade F11 blockedFastly
983mailchi.mpScore 6 out of 100, grade F, partial assessment11 blockedunidentified
987btloader.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
989n-tv.deScore 52 out of 100, grade D5 blockedNext.js . Amazon CloudFront

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking Applebot-Extended." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.