crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

ByteDance . tier 2

Bytespider

Collects training data for ByteDance models.

551 of the 3,656 sites we measured deny Bytespider access to their root, which is 15.1% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow Bytespider everywhere
User-agent: Bytespider
Disallow:

# Block Bytespider entirely
User-agent: Bytespider
Disallow: /

Sites blocking Bytespider

Most-visited first. Showing up to 100 of 551.

Domains that block Bytespider
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
29amazon.comrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
34github.comScore 72 out of 100, grade Call allowedContentful
43netflix.comScore 59 out of 100, grade D, partial assessment3 blockedunidentified
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
63tiktok.comScore 15 out of 100, grade F11 blockedunidentified
72yahoo.comrefused GPTBotScore 43 out of 100, grade D5 blockedNext.js
81amazonvideo.comScore 40 out of 100, grade D5 blockedAmazon CloudFront
84vimeo.comScore 56 out of 100, grade D3 blockedNext.js . Cloudflare
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
96t-online.deScore 55 out of 100, grade D5 blockedNext.js . Amazon CloudFront
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
155nytimes.comrefused GPTBotScore 21 out of 100, grade F, partial assessment11 blockedFastly
159snapchat.comScore 46 out of 100, grade D5 blockedNext.js . Google Cloud / GFE
167gravatar.comScore 60 out of 100, grade C2 blockedWordPress
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
183forter.comScore 87 out of 100, grade B1 blockedWordPress . Cloudflare
192cnn.comScore 31 out of 100, grade F9 blockedFastly
198ebay.comScore 60 out of 100, grade C, partial assessment5 blockedAkamai
202soundcloud.comScore 31 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
203theguardian.comScore 46 out of 100, grade D6 blockedunidentified
206unity3d.comScore 79 out of 100, grade Ball allowedSanity
209medium.comrefused GPTBotScore 46 out of 100, grade D4 blockedCloudflare
219bbc.comScore 31 out of 100, grade F10 blockedNext.js . Fastly
226weather.comScore 82 out of 100, grade Ball allowedNext.js . Akamai
227forbes.comrefused GPTBotScore 57 out of 100, grade D, partial assessment5 blockedFastly
242bbc.co.ukScore 34 out of 100, grade F10 blockedFastly
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
268linktr.eeScore 81 out of 100, grade B, partial assessment1 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
294nature.comScore 45 out of 100, grade D, partial assessment7 blockedFastly
296launchpad.netrefused GPTBotScore 20 out of 100, grade F9 blockedunidentified
298canva.comrefused GPTBotScore 66 out of 100, grade C, partial assessment4 blockedCloudflare
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
332nflxvideo.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
333amazon.co.ukrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
340espn.comScore 72 out of 100, grade C, partial assessment3 blockedAmazon CloudFront
341trustpilot.comScore 66 out of 100, grade C, partial assessment2 blockedunidentified
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
353ea.comScore 66 out of 100, grade C1 blockedNext.js
371wsj.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
386amazon.derefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
404bloomberg.comrefused GPTBotScore 27 out of 100, grade F, partial assessment10 blockedFastly
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
429temu.comScore 74 out of 100, grade Call allowedunidentified
430aliexpress.comScore 71 out of 100, grade C, partial assessment3 blockedAlibaba Cloud CDN
433aol.comScore 50 out of 100, grade D3 blockedNext.js . Amazon CloudFront
448dailymotion.comScore 57 out of 100, grade Dall allowedAmazon CloudFront
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
467pixabay.comrefused GPTBotScore 67 out of 100, grade C, partial assessment4 blockedCloudflare
486cnbc.comrefused GPTBotScore 24 out of 100, grade F11 blockedAkamai
492businessinsider.comScore 72 out of 100, grade C2 blockedFastly
495amazon.co.jprefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
504quora.comScore 13 out of 100, grade F, partial assessment10 blockedunidentified
509statista.comrefused GPTBotScore 43 out of 100, grade D7 blockedAmazon CloudFront
536behance.netrefused GPTBotScore 60 out of 100, grade C, partial assessment5 blockedFastly
555ft.comrefused GPTBotScore 53 out of 100, grade D, partial assessment6 blockedCloudflare
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
572wired.comScore 51 out of 100, grade D8 blockedCloudflare
582dailymail.comrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
600uol.com.brScore 47 out of 100, grade D8 blockedAmazon CloudFront
601tripadvisor.comrefused GPTBotScore 63 out of 100, grade C, partial assessment4 blockedAmazon CloudFront
622techcrunch.comScore 66 out of 100, grade C3 blockedWordPress . Cloudflare
629usatoday.comScore 16 out of 100, grade F, partial assessment11 blockedunidentified
630telegraph.co.ukScore 37 out of 100, grade F10 blockedAdobe Experience Manager . Fastly
632amazon.frrefused GPTBotScore 15 out of 100, grade F10 blockedAmazon CloudFront
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
644amazon.carefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
668nintendo.comScore 48 out of 100, grade D5 blockedNext.js . Fastly
674elpais.comrefused GPTBotScore 49 out of 100, grade D, partial assessment5 blockedunidentified
675amazon.inrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
690spiegel.deScore 40 out of 100, grade D8 blockedGoogle Cloud / GFE
692kick.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
695lemonde.frScore 48 out of 100, grade D, partial assessment5 blockedFastly
700bandcamp.comScore 46 out of 100, grade D4 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
707globalsign.comScore 56 out of 100, grade D5 blockedFastly
708hugedomains.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
720amazon.esrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
730cookielaw.orgrefused GPTBotScore 51 out of 100, grade D5 blockedAdobe Experience Manager . Cloudflare
734dailymail.co.ukrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
736amazon.itrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
749netflix.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
756cnet.comScore 44 out of 100, grade D10 blockedWordPress
758bild.deScore 59 out of 100, grade D3 blockedunidentified
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
786welt.deScore 50 out of 100, grade D3 blockedunidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking Bytespider." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.