crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Google . tier 1

Google-Extended

Controls use in Gemini training and grounding. Does not affect Google Search ranking.

459 of the 3,656 sites we measured deny Google-Extended access to their root, which is 12.6% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow Google-Extended everywhere
User-agent: Google-Extended
Disallow:

# Block Google-Extended entirely
User-agent: Google-Extended
Disallow: /

Sites blocking Google-Extended

Most-visited first. Showing up to 100 of 459.

Domains that block Google-Extended
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
29amazon.comrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
63tiktok.comScore 15 out of 100, grade F11 blockedunidentified
72yahoo.comrefused GPTBotScore 43 out of 100, grade D5 blockedNext.js
84vimeo.comScore 56 out of 100, grade D3 blockedNext.js . Cloudflare
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
153tumblr.comScore 54 out of 100, grade D4 blockedWordPress
155nytimes.comrefused GPTBotScore 21 out of 100, grade F, partial assessment11 blockedFastly
159snapchat.comScore 46 out of 100, grade D5 blockedNext.js . Google Cloud / GFE
167gravatar.comScore 60 out of 100, grade C2 blockedWordPress
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
202soundcloud.comScore 31 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
219bbc.comScore 31 out of 100, grade F10 blockedNext.js . Fastly
242bbc.co.ukScore 34 out of 100, grade F10 blockedFastly
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
294nature.comScore 45 out of 100, grade D, partial assessment7 blockedFastly
296launchpad.netrefused GPTBotScore 20 out of 100, grade F9 blockedunidentified
327alibaba.comScore 53 out of 100, grade D2 blockedAlibaba Cloud CDN
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
333amazon.co.ukrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
340espn.comScore 72 out of 100, grade C, partial assessment3 blockedAmazon CloudFront
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
371wsj.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
386amazon.derefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
392globo.comScore 56 out of 100, grade D2 blockedunidentified
393go.comScore 56 out of 100, grade D2 blockedAmazon CloudFront
404bloomberg.comrefused GPTBotScore 27 out of 100, grade F, partial assessment10 blockedFastly
413claude.airefused GPTBotScore 51 out of 100, grade D4 blockedCloudflare
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
430aliexpress.comScore 71 out of 100, grade C, partial assessment3 blockedAlibaba Cloud CDN
433aol.comScore 50 out of 100, grade D3 blockedNext.js . Amazon CloudFront
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
462yahoo.co.jpScore 55 out of 100, grade D2 blockedunidentified
467pixabay.comrefused GPTBotScore 67 out of 100, grade C, partial assessment4 blockedCloudflare
486cnbc.comrefused GPTBotScore 24 out of 100, grade F11 blockedAkamai
495amazon.co.jprefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
509statista.comrefused GPTBotScore 43 out of 100, grade D7 blockedAmazon CloudFront
536behance.netrefused GPTBotScore 60 out of 100, grade C, partial assessment5 blockedFastly
555ft.comrefused GPTBotScore 53 out of 100, grade D, partial assessment6 blockedCloudflare
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
569sohu.comScore 8 out of 100, grade F, partial assessment11 blockedunidentified
572wired.comScore 51 out of 100, grade D8 blockedCloudflare
582dailymail.comrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
600uol.com.brScore 47 out of 100, grade D8 blockedAmazon CloudFront
629usatoday.comScore 16 out of 100, grade F, partial assessment11 blockedunidentified
632amazon.frrefused GPTBotScore 15 out of 100, grade F10 blockedAmazon CloudFront
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
644amazon.carefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
668nintendo.comScore 48 out of 100, grade D5 blockedNext.js . Fastly
675amazon.inrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
692kick.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
695lemonde.frScore 48 out of 100, grade D, partial assessment5 blockedFastly
700bandcamp.comScore 46 out of 100, grade D4 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
707globalsign.comScore 56 out of 100, grade D5 blockedFastly
708hugedomains.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
720amazon.esrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
730cookielaw.orgrefused GPTBotScore 51 out of 100, grade D5 blockedAdobe Experience Manager . Cloudflare
734dailymail.co.ukrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
736amazon.itrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
800sharethrough.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
811attn.tvScore 6 out of 100, grade F, partial assessment11 blockedunidentified
825people.comScore 35 out of 100, grade F, partial assessment8 blockedCloudflare
838ted.comScore 52 out of 100, grade D5 blockedNext.js . Fastly
845salesforceliveagent.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
849adform.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
853theverge.comScore 44 out of 100, grade D9 blockedWordPress . Fastly
856academia.eduScore 48 out of 100, grade D, partial assessment6 blockedunidentified
868imgsmail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
882nbcnews.comScore 30 out of 100, grade F11 blockedNext.js . Akamai
887amazon.com.brrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
907figma.comScore 47 out of 100, grade D8 blockedSanity . Amazon CloudFront
914hbr.orgrefused GPTBotScore 38 out of 100, grade F5 blockedNext.js . Amazon CloudFront
935daum.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
939repubblica.itScore 65 out of 100, grade C4 blockedAmazon CloudFront
954huffpost.comScore 46 out of 100, grade D11 blockedFastly
955corriere.itrefused GPTBotScore 34 out of 100, grade F11 blockedFastly
964amazon.com.aurefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
983mailchi.mpScore 6 out of 100, grade F, partial assessment11 blockedunidentified
987btloader.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
989n-tv.deScore 52 out of 100, grade D5 blockedNext.js . Amazon CloudFront
1010themoviedb.orgrefused GPTBotScore 18 out of 100, grade F, partial assessment11 blockedAmazon CloudFront
1017pixiv.netrefused GPTBotScore 49 out of 100, grade D, partial assessment5 blockedCloudflare

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking Google-Extended." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.