Tavily . tier 2

TavilyBot

Retrieval API used by agent frameworks.

191 of the 3,618 sites we measured deny TavilyBot access to their root, which is 5.3% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow TavilyBot everywhere
User-agent: TavilyBot
Disallow:

# Block TavilyBot entirely
User-agent: TavilyBot
Disallow: /

Sites blocking TavilyBot

Most-visited first. Showing up to 100 of 191.

Domains that block TavilyBot
Rank	Domain	Score	Answer-surface crawlers	Stack
3	facebook.com	Score 35 out of 100, grade F, partial assessment	6 blocked	unidentified
11	instagram.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
12	mail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
15	twitter.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
17	linkedin.com	Score 17 out of 100, grade F, partial assessment	9 blocked	unidentified
25	amazon.com	Score 16 out of 100, grade F, partial assessment	10 blocked	Akamai
41	netflix.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
49	x.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
56	pinterest.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
71	myfritz.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
76	chatgpt.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
87	baidu.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
109	reddit.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
122	amazonvideo.com	Score 37 out of 100, grade F	5 blocked	Amazon CloudFront
190	flickr.com	Score 42 out of 100, grade D, partial assessment	6 blocked	unidentified
218	bbc.com	Score 34 out of 100, grade F	9 blocked	Next.js . Fastly
236	bbc.co.uk	Score 37 out of 100, grade F	9 blocked	Fastly
249	meraki.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
268	imdb.com	Score 9 out of 100, grade F, partial assessment	11 blocked	unidentified
272	linktr.ee	Score 81 out of 100, grade B, partial assessment	1 blocked	unidentified
287	reuters.com	Score 26 out of 100, grade F, partial assessment	9 blocked	unidentified
307	weibo.com	Score 18 out of 100, grade F, partial assessment	11 blocked	unidentified
335	amazon.co.uk	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
350	trustpilot.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
353	naver.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
380	wsj.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
391	amazon.de	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
411	nflxvideo.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
459	gcdn.co	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
462	edgecdn.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
477	sohu.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
483	amazon.co.za	Score 16 out of 100, grade F, partial assessment	10 blocked	Akamai
488	skybridge.click	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
495	amazon.co.jp	Score 16 out of 100, grade F, partial assessment	10 blocked	Akamai
499	quora.com	Score 13 out of 100, grade F, partial assessment	10 blocked	unidentified
550	netflix.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
586	usatoday.com	Score 24 out of 100, grade F	11 blocked	unidentified
600	imgsmail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
617	threads.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
626	amazon.fr	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
637	sharethrough.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
647	amazon.ca	Score 22 out of 100, grade F	10 blocked	Amazon CloudFront
664	yelp.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
668	hstgr.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
673	primevideo.com	Score 37 out of 100, grade F	5 blocked	Amazon CloudFront
687	hihonorcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
690	amazon.in	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
698	vkontakte.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
707	lemonde.fr	Score 49 out of 100, grade D	5 blocked	Fastly
717	fb.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
720	amazon.es	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
721	pinimg.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
725	amazon.it	Score 16 out of 100, grade F, partial assessment	10 blocked	Amazon CloudFront
730	quantserve.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
732	cnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
741	creativecdn.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
809	disneyplus.com	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
817	adform.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
837	lefigaro.fr	Score 55 out of 100, grade D	7 blocked	unidentified
843	daum.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
849	amazon.com.au	Score 15 out of 100, grade F	10 blocked	Amazon CloudFront
862	amazon.com.br	Score 16 out of 100, grade F, partial assessment	10 blocked	Akamai
963	stbid.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
965	blackhub.team	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
969	mailchi.mp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
987	btloader.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1040	amazon.com.mxrefused GPTBot	Score 15 out of 100, grade F	10 blocked	Amazon CloudFront
1069	ouest-france.fr	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
1071	rocketmoney.dev	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1113	theconversation.com	Score 26 out of 100, grade F	11 blocked	Next.js . Fastly
1118	xiaohongshu.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1138	netcraze.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1196	fmkorea.com	Score 34 out of 100, grade F, partial assessment	6 blocked	unidentified
1228	namu.wiki	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1236	namecheaphosting.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1237	kakao.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1247	amap.com	Score 54 out of 100, grade D, partial assessment	5 blocked	unidentified
1254	furaffinity.netrefused GPTBot	Score 6 out of 100, grade F, partial assessment	11 blocked	Cloudflare
1515	exhentai.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1522	attn.tv	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1548	tjk.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1571	aka.ms	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1663	sportfilm800.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1670	keirin.jp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1752	bpexch.live	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1782	threads.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1790	qcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1821	avito.st	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1858	zdnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
1864	marketwatch.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
1868	adrta.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1875	mega.nz	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1939	franceinfo.fr	Score 27 out of 100, grade F	11 blocked	unidentified
1960	sedoparking.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1976	secureserver.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1982	clevertap-prod.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2008	vtb.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2020	sogou.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2021	ign.comrefused GPTBot	Score 33 out of 100, grade F	10 blocked	unidentified
2052	salesforceliveagent.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking TavilyBot." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.