crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Kagi . tier 2

kagi-fetcher

Fetches pages for Kagi assistant features.

167 of the 3,656 sites we measured deny kagi-fetcher access to their root, which is 4.6% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow kagi-fetcher everywhere
User-agent: kagi-fetcher
Disallow:

# Block kagi-fetcher entirely
User-agent: kagi-fetcher
Disallow: /

Sites blocking kagi-fetcher

Most-visited first. Showing up to 100 of 167.

Domains that block kagi-fetcher
RankDomainScoreAnswer-surface crawlersStack
5facebook.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
14instagram.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
16twitter.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
17linkedin.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
27mail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
43netflix.comScore 59 out of 100, grade D, partial assessment3 blockedunidentified
48x.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
55pinterest.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
62myfritz.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
90baidu.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
111reddit.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
114chatgpt.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
170skybridge.clickScore 6 out of 100, grade F, partial assessment11 blockedunidentified
181flickr.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified
265imdb.comScore 9 out of 100, grade F, partial assessment11 blockedunidentified
268linktr.eeScore 81 out of 100, grade B, partial assessment1 blockedunidentified
285reuters.comScore 26 out of 100, grade F, partial assessment9 blockedunidentified
331weibo.comScore 18 out of 100, grade F, partial assessment11 blockedunidentified
332nflxvideo.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
341trustpilot.comScore 66 out of 100, grade C, partial assessment2 blockedunidentified
351naver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
371wsj.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
425gcdn.coScore 6 out of 100, grade F, partial assessment11 blockedunidentified
458netcraze.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
504quora.comScore 13 out of 100, grade F, partial assessment10 blockedunidentified
565threads.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
569sohu.comScore 8 out of 100, grade F, partial assessment11 blockedunidentified
636edgecdn.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
667hstgr.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
695lemonde.frScore 48 out of 100, grade D, partial assessment5 blockedFastly
702fb.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
706yelp.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
724quantserve.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
729pinimg.comScore 17 out of 100, grade F, partial assessment11 blockedunidentified
741creativecdn.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
749netflix.netScore 59 out of 100, grade D, partial assessment3 blockedunidentified
756cnet.comScore 44 out of 100, grade D10 blockedWordPress
762hihonorcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
800sharethrough.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
811attn.tvScore 6 out of 100, grade F, partial assessment11 blockedunidentified
824disneyplus.comScore 77 out of 100, grade B, partial assessment1 blockedunidentified
845salesforceliveagent.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
849adform.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
856academia.eduScore 48 out of 100, grade D, partial assessment6 blockedunidentified
868imgsmail.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
935daum.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
983mailchi.mpScore 6 out of 100, grade F, partial assessment11 blockedunidentified
987btloader.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1027xiaohongshu.comScore 14 out of 100, grade F, partial assessment11 blockedunidentified
1038ouest-france.frScore 14 out of 100, grade F, partial assessment11 blockedunidentified
1159namecheaphosting.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1195theconversation.comScore 26 out of 100, grade F11 blockedNext.js . Fastly
1196namu.wikiScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1217fmkorea.comScore 34 out of 100, grade F, partial assessment6 blockedunidentified
1219furaffinity.netrefused GPTBotScore 6 out of 100, grade F, partial assessment11 blockedCloudflare
1227amap.comScore 54 out of 100, grade D, partial assessment5 blockedunidentified
1296kakao.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1307vkontakte.ruScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1387aternos.orgScore 14 out of 100, grade F, partial assessment11 blockedunidentified
1510exhentai.orgScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1538aka.msScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1602tjk.orgScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1680keirin.jpScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1720threads.netScore 14 out of 100, grade F, partial assessment11 blockedunidentified
1759bpexch.xyzScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1765ind.xxxScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1795qcloud.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1803tappx.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1844avito.stScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1874hulu.comScore 77 out of 100, grade B, partial assessment1 blockedunidentified
1885zdnet.comScore 44 out of 100, grade D10 blockedWordPress
1894marketwatch.comScore 31 out of 100, grade F, partial assessment8 blockedunidentified
1911adrta.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1914clevertap-prod.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
1918mega.nzScore 14 out of 100, grade F, partial assessment11 blockedunidentified
1996kasserver.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2040salesforce-scrt.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2148ign.comrefused GPTBotScore 29 out of 100, grade F10 blockedWordPress . Fastly
2192sap.comScore 68 out of 100, grade C, partial assessment2 blockedunidentified
2207thesun.co.ukScore 31 out of 100, grade F, partial assessment8 blockedunidentified
2216mashable.comrefused GPTBotScore 35 out of 100, grade F10 blockedCloudflare
2220sogou.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2264permutive.appScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2273openssl.orgScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2309ad.gtScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2337weibo.cnScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2375secureserver.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2413pxf.ioScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2453zeit.deScore 16 out of 100, grade F, partial assessment11 blockedunidentified
2498instacart.comScore 17 out of 100, grade F, partial assessment9 blockedunidentified
2513redditmedia.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2558pcmag.comrefused GPTBotScore 35 out of 100, grade F10 blockedCloudflare
2563news.com.auScore 14 out of 100, grade F, partial assessment11 blockedunidentified
2564system-monitor.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2568expressapisv2.netScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2594nextdoor.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2624siteforce.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2632kahoot.itScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2644urldefense.comScore 6 out of 100, grade F, partial assessment11 blockedunidentified
2654kuaishou.comScore 42 out of 100, grade D, partial assessment6 blockedunidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking kagi-fetcher." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.