Kagi . tier 2

kagi-fetcher

Fetches pages for Kagi assistant features.

158 of the 3,618 sites we measured deny kagi-fetcher access to their root, which is 4.4% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow kagi-fetcher everywhere
User-agent: kagi-fetcher
Disallow:

# Block kagi-fetcher entirely
User-agent: kagi-fetcher
Disallow: /

Sites blocking kagi-fetcher

Most-visited first. Showing up to 100 of 158.

Domains that block kagi-fetcher
Rank	Domain	Score	Answer-surface crawlers	Stack
3	facebook.com	Score 35 out of 100, grade F, partial assessment	6 blocked	unidentified
11	instagram.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
12	mail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
15	twitter.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
17	linkedin.com	Score 17 out of 100, grade F, partial assessment	9 blocked	unidentified
41	netflix.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
49	x.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
56	pinterest.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
71	myfritz.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
76	chatgpt.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
87	baidu.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
109	reddit.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
190	flickr.com	Score 42 out of 100, grade D, partial assessment	6 blocked	unidentified
249	meraki.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
268	imdb.com	Score 9 out of 100, grade F, partial assessment	11 blocked	unidentified
272	linktr.ee	Score 81 out of 100, grade B, partial assessment	1 blocked	unidentified
287	reuters.com	Score 26 out of 100, grade F, partial assessment	9 blocked	unidentified
307	weibo.com	Score 18 out of 100, grade F, partial assessment	11 blocked	unidentified
350	trustpilot.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
353	naver.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
380	wsj.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
411	nflxvideo.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
459	gcdn.co	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
462	edgecdn.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
477	sohu.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
488	skybridge.click	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
499	quora.com	Score 13 out of 100, grade F, partial assessment	10 blocked	unidentified
550	netflix.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
600	imgsmail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
617	threads.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
637	sharethrough.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
664	yelp.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
668	hstgr.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
687	hihonorcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
698	vkontakte.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
707	lemonde.fr	Score 49 out of 100, grade D	5 blocked	Fastly
717	fb.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
721	pinimg.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
730	quantserve.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
732	cnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
741	creativecdn.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
809	disneyplus.com	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
817	adform.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
836	academia.edu	Score 48 out of 100, grade D, partial assessment	6 blocked	unidentified
843	daum.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
963	stbid.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
965	blackhub.team	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
969	mailchi.mp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
987	btloader.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1069	ouest-france.fr	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
1071	rocketmoney.dev	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1113	theconversation.com	Score 26 out of 100, grade F	11 blocked	Next.js . Fastly
1118	xiaohongshu.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1138	netcraze.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1196	fmkorea.com	Score 34 out of 100, grade F, partial assessment	6 blocked	unidentified
1228	namu.wiki	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1236	namecheaphosting.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1237	kakao.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1247	amap.com	Score 54 out of 100, grade D, partial assessment	5 blocked	unidentified
1254	furaffinity.netrefused GPTBot	Score 6 out of 100, grade F, partial assessment	11 blocked	Cloudflare
1515	exhentai.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1522	attn.tv	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1548	tjk.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1571	aka.ms	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1663	sportfilm800.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1670	keirin.jp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1752	bpexch.live	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1782	threads.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1790	qcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1821	avito.st	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1858	zdnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
1864	marketwatch.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
1868	adrta.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1875	mega.nz	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1960	sedoparking.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1976	secureserver.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1982	clevertap-prod.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2008	vtb.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2020	sogou.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2052	salesforceliveagent.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2068	mashable.comrefused GPTBot	Score 35 out of 100, grade F	10 blocked	Cloudflare
2083	thesun.co.uk	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
2108	sureserver.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2112	openssl.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2113	sap.com	Score 68 out of 100, grade C, partial assessment	2 blocked	unidentified
2162	ad.gt	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2269	juicyscore.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
2306	co.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2344	weibo.cn	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2399	redditmedia.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2493	merkur.de	Score 48 out of 100, grade D, partial assessment	5 blocked	unidentified
2531	news.com.au	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
2581	expressapisv2.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2593	pxf.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2596	urldefense.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2626	kuaishou.com	Score 42 out of 100, grade D, partial assessment	6 blocked	unidentified
2640	system-monitor.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2671	instacart.com	Score 17 out of 100, grade F, partial assessment	9 blocked	unidentified
2735	huntress.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2745	ksapisrv.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking kagi-fetcher." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.