crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Cohere . tier 2

cohere-ai

Retrieval for Cohere assistants.

451 of the 3,656 sites we measured deny cohere-ai access to their root, which is 12.3% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow cohere-ai everywhere
User-agent: cohere-ai
Disallow:

# Block cohere-ai entirely
User-agent: cohere-ai
Disallow: /

Sites blocking cohere-ai

Most-visited first. Showing up to 100 of 451.

Domains that block cohere-ai
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
29amazon.comrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
43netflix.comScore 59 out of 100, grade D, partial assessment3 blockedunidentified
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
72yahoo.comrefused GPTBotScore 43 out of 100, grade D5 blockedNext.js
81amazonvideo.comScore 40 out of 100, grade D5 blockedAmazon CloudFront
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
155nytimes.comrefused GPTBotScore 21 out of 100, grade F, partial assessment11 blockedFastly
159snapchat.comScore 46 out of 100, grade D5 blockedNext.js . Google Cloud / GFE
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
192cnn.comScore 31 out of 100, grade F9 blockedFastly
202soundcloud.comScore 31 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
219bbc.comScore 31 out of 100, grade F10 blockedNext.js . Fastly
227forbes.comrefused GPTBotScore 57 out of 100, grade D, partial assessment5 blockedFastly
242bbc.co.ukScore 34 out of 100, grade F10 blockedFastly
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
268linktr.eeScore 81 out of 100, grade B, partial assessment1 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
294nature.comScore 45 out of 100, grade D, partial assessment7 blockedFastly
296launchpad.netrefused GPTBotScore 20 out of 100, grade F9 blockedunidentified
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
332nflxvideo.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
333amazon.co.ukrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
341trustpilot.comScore 66 out of 100, grade C, partial assessment2 blockedunidentified
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
353ea.comScore 66 out of 100, grade C1 blockedNext.js
371wsj.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
386amazon.derefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
404bloomberg.comrefused GPTBotScore 27 out of 100, grade F, partial assessment10 blockedFastly
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
433aol.comScore 50 out of 100, grade D3 blockedNext.js . Amazon CloudFront
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
486cnbc.comrefused GPTBotScore 24 out of 100, grade F11 blockedAkamai
495amazon.co.jprefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
504quora.comScore 13 out of 100, grade F, partial assessment10 blockedunidentified
509statista.comrefused GPTBotScore 43 out of 100, grade D7 blockedAmazon CloudFront
555ft.comrefused GPTBotScore 53 out of 100, grade D, partial assessment6 blockedCloudflare
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
569sohu.comScore 8 out of 100, grade F, partial assessment11 blockedunidentified
572wired.comScore 51 out of 100, grade D8 blockedCloudflare
582dailymail.comrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
600uol.com.brScore 47 out of 100, grade D8 blockedAmazon CloudFront
601tripadvisor.comrefused GPTBotScore 63 out of 100, grade C, partial assessment4 blockedAmazon CloudFront
622techcrunch.comScore 66 out of 100, grade C3 blockedWordPress . Cloudflare
629usatoday.comScore 16 out of 100, grade F, partial assessment11 blockedunidentified
630telegraph.co.ukScore 37 out of 100, grade F10 blockedAdobe Experience Manager . Fastly
632amazon.frrefused GPTBotScore 15 out of 100, grade F10 blockedAmazon CloudFront
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
644amazon.carefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
668nintendo.comScore 48 out of 100, grade D5 blockedNext.js . Fastly
675amazon.inrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
690spiegel.deScore 40 out of 100, grade D8 blockedGoogle Cloud / GFE
692kick.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
695lemonde.frScore 48 out of 100, grade D, partial assessment5 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
708hugedomains.comrefused GPTBotScore 61 out of 100, grade C, partial assessment5 blockedCloudflare
720amazon.esrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
730cookielaw.orgrefused GPTBotScore 51 out of 100, grade D5 blockedAdobe Experience Manager . Cloudflare
734dailymail.co.ukrefused GPTBotScore 35 out of 100, grade F11 blockedAkamai
736amazon.itrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
749netflix.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
756cnet.comScore 44 out of 100, grade D10 blockedWordPress
758bild.deScore 59 out of 100, grade D3 blockedunidentified
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
770apnews.comrefused GPTBotScore 49 out of 100, grade D, partial assessment7 blockedCloudflare
786welt.deScore 50 out of 100, grade D3 blockedunidentified
800sharethrough.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
811attn.tvScore 6 out of 100, grade F, partial assessment11 blockedunidentified
824disneyplus.comScore 77 out of 100, grade B, partial assessment1 blockedunidentified
825people.comScore 35 out of 100, grade F, partial assessment8 blockedCloudflare
838ted.comScore 52 out of 100, grade D5 blockedNext.js . Fastly
845salesforceliveagent.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
849adform.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
853theverge.comScore 44 out of 100, grade D9 blockedWordPress . Fastly
856academia.eduScore 48 out of 100, grade D, partial assessment6 blockedunidentified
868imgsmail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
882nbcnews.comScore 30 out of 100, grade F11 blockedNext.js . Akamai
887amazon.com.brrefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront
907figma.comScore 47 out of 100, grade D8 blockedSanity . Amazon CloudFront
913pornhub.comScore 69 out of 100, grade Call allowedunidentified
935daum.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
939repubblica.itScore 65 out of 100, grade C4 blockedAmazon CloudFront
954huffpost.comScore 46 out of 100, grade D11 blockedFastly
955corriere.itrefused GPTBotScore 34 out of 100, grade F11 blockedFastly
956onet.plScore 70 out of 100, grade C1 blockedNext.js . Amazon CloudFront
964amazon.com.aurefused GPTBotScore 16 out of 100, grade F, partial assessment10 blockedAmazon CloudFront

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking cohere-ai." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.