Firecrawl . tier 2

FirecrawlAgent

Retrieval API used by agent frameworks.

218 of the 3,642 sites we measured deny FirecrawlAgent access to their root, which is 6.0% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow FirecrawlAgent everywhere
User-agent: FirecrawlAgent
Disallow:

# Block FirecrawlAgent entirely
User-agent: FirecrawlAgent
Disallow: /

Sites blocking FirecrawlAgent

Most-visited first. Showing up to 100 of 218.

Domains that block FirecrawlAgent
Rank	Domain	Score	Answer-surface crawlers	Stack
5	facebook.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
14	instagram.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
16	twitter.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
17	linkedin.com	Score 17 out of 100, grade F, partial assessment	9 blocked	unidentified
18	mail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
43	netflix.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
47	x.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
52	pinterest.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
61	myfritz.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
85	baidu.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
103	reddit.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
112	chatgpt.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
126	skybridge.click	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
163	nytimes.comrefused GPTBot	Score 21 out of 100, grade F, partial assessment	11 blocked	Fastly
180	flickr.com	Score 42 out of 100, grade D, partial assessment	6 blocked	unidentified
193	cnn.com	Score 31 out of 100, grade F	9 blocked	Fastly
218	bbc.com	Score 31 out of 100, grade F	10 blocked	Next.js . Fastly
240	bbc.co.uk	Score 34 out of 100, grade F	10 blocked	Fastly
263	imdb.com	Score 9 out of 100, grade F, partial assessment	11 blocked	unidentified
264	linktr.ee	Score 81 out of 100, grade B, partial assessment	1 blocked	unidentified
283	reuters.com	Score 26 out of 100, grade F, partial assessment	9 blocked	unidentified
284	launchpad.netrefused GPTBot	Score 20 out of 100, grade F	9 blocked	unidentified
325	weibo.com	Score 18 out of 100, grade F, partial assessment	11 blocked	unidentified
344	naver.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
353	trustpilot.com	Score 66 out of 100, grade C, partial assessment	2 blocked	unidentified
359	nflxvideo.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
372	wsj.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
421	netflix.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
439	gcdn.co	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
476	netcraze.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
497	quora.com	Score 13 out of 100, grade F, partial assessment	10 blocked	unidentified
566	sohu.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
574	wired.com	Score 51 out of 100, grade D	8 blocked	Cloudflare
580	edgecdn.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
589	threads.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
622	usatoday.com	Score 24 out of 100, grade F	11 blocked	Sitecore
655	hstgr.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
692	sharethrough.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
699	lemonde.fr	Score 48 out of 100, grade D, partial assessment	5 blocked	Fastly
703	yelp.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
708	fb.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
730	pinimg.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
733	hihonorcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
737	quantserve.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
749	cnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
757	creativecdn.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
762	bild.de	Score 59 out of 100, grade D	3 blocked	unidentified
765	imgsmail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
787	welt.de	Score 50 out of 100, grade D	3 blocked	unidentified
820	disneyplus.com	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
825	people.com	Score 35 out of 100, grade F, partial assessment	8 blocked	Cloudflare
850	adform.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
852	academia.edu	Score 48 out of 100, grade D, partial assessment	6 blocked	unidentified
866	theverge.com	Score 44 out of 100, grade D	9 blocked	WordPress . Fastly
885	daum.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
934	xiaohongshu.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
980	vkontakte.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
986	mailchi.mp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
997	btloader.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1016	namecheaphosting.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1037	attn.tv	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1055	ouest-france.fr	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1059	investopedia.com	Score 35 out of 100, grade F, partial assessment	8 blocked	Cloudflare
1068	themoviedb.orgrefused GPTBot	Score 18 out of 100, grade F, partial assessment	11 blocked	Amazon CloudFront
1070	bfmtv.com	Score 35 out of 100, grade F	11 blocked	Amazon CloudFront
1084	redfin.com	Score 90 out of 100, grade A, partial assessment	all allowed	Amazon CloudFront
1120	rocketmoney.dev	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1186	theconversation.com	Score 26 out of 100, grade F	11 blocked	Next.js . Fastly
1190	namu.wiki	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1218	fmkorea.com	Score 34 out of 100, grade F, partial assessment	6 blocked	unidentified
1242	amap.com	Score 54 out of 100, grade D, partial assessment	5 blocked	unidentified
1245	furaffinity.netrefused GPTBot	Score 6 out of 100, grade F, partial assessment	11 blocked	Cloudflare
1297	kakao.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1393	aternos.org	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1442	avito.st	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1459	ptt.ccrefused GPTBot	Score 32 out of 100, grade F, partial assessment	8 blocked	Cloudflare
1523	exhentai.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1557	aka.ms	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1596	tjk.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1641	twkan.comrefused GPTBot	Score 32 out of 100, grade F, partial assessment	8 blocked	Cloudflare
1655	darkero.comrefused GPTBot	Score 45 out of 100, grade D	8 blocked	WordPress . Cloudflare
1678	keirin.jp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1692	nontondrama.myrefused GPTBot	Score 32 out of 100, grade F, partial assessment	8 blocked	Cloudflare
1748	threads.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1757	ind.xxx	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1799	qcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1807	stbid.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1809	lefigaro.fr	Score 55 out of 100, grade D	7 blocked	unidentified
1827	newyorker.com	Score 51 out of 100, grade D	8 blocked	Amazon CloudFront
1863	zdnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
1881	marketwatch.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
1889	hulu.com	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
1891	mega.nz	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1896	adrta.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1901	salesforceliveagent.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1937	clevertap-prod.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1943	weibo.cn	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1954	jamanetwork.comrefused GPTBot	Score 57 out of 100, grade D, partial assessment	5 blocked	Cloudflare
1959	wikihow.com	Score 21 out of 100, grade F, partial assessment	9 blocked	unidentified
2041	ign.comrefused GPTBot	Score 26 out of 100, grade F	10 blocked	WordPress . Fastly

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking FirecrawlAgent." https://crawlindex.org (measured 2026-11-07). Licensed CC BY 4.0.