Firecrawl . tier 2

FirecrawlAgent

Retrieval API used by agent frameworks.

207 of the 3,618 sites we measured deny FirecrawlAgent access to their root, which is 5.7% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow FirecrawlAgent everywhere
User-agent: FirecrawlAgent
Disallow:

# Block FirecrawlAgent entirely
User-agent: FirecrawlAgent
Disallow: /

Sites blocking FirecrawlAgent

Most-visited first. Showing up to 100 of 207.

Domains that block FirecrawlAgent
Rank	Domain	Score	Answer-surface crawlers	Stack
3	facebook.com	Score 35 out of 100, grade F, partial assessment	6 blocked	unidentified
11	instagram.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
12	mail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
15	twitter.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
17	linkedin.com	Score 17 out of 100, grade F, partial assessment	9 blocked	unidentified
41	netflix.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
49	x.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
56	pinterest.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
71	myfritz.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
76	chatgpt.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
87	baidu.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
109	reddit.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
159	nytimes.comrefused GPTBot	Score 21 out of 100, grade F, partial assessment	11 blocked	Fastly
190	flickr.com	Score 42 out of 100, grade D, partial assessment	6 blocked	unidentified
218	bbc.com	Score 34 out of 100, grade F	9 blocked	Next.js . Fastly
236	bbc.co.uk	Score 37 out of 100, grade F	9 blocked	Fastly
249	meraki.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
268	imdb.com	Score 9 out of 100, grade F, partial assessment	11 blocked	unidentified
272	linktr.ee	Score 81 out of 100, grade B, partial assessment	1 blocked	unidentified
277	launchpad.netrefused GPTBot	Score 20 out of 100, grade F	9 blocked	unidentified
287	reuters.com	Score 26 out of 100, grade F, partial assessment	9 blocked	unidentified
307	weibo.com	Score 18 out of 100, grade F, partial assessment	11 blocked	unidentified
350	trustpilot.com	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
353	naver.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
380	wsj.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
411	nflxvideo.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
459	gcdn.co	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
462	edgecdn.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
477	sohu.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
488	skybridge.click	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
499	quora.com	Score 13 out of 100, grade F, partial assessment	10 blocked	unidentified
550	netflix.net	Score 59 out of 100, grade D, partial assessment	3 blocked	unidentified
554	wired.com	Score 51 out of 100, grade D	8 blocked	Cloudflare
586	usatoday.com	Score 24 out of 100, grade F	11 blocked	unidentified
600	imgsmail.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
617	threads.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
637	sharethrough.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
664	yelp.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
668	hstgr.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
687	hihonorcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
698	vkontakte.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
707	lemonde.fr	Score 49 out of 100, grade D	5 blocked	Fastly
717	fb.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
721	pinimg.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
730	quantserve.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
732	cnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
741	creativecdn.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
766	bild.de	Score 62 out of 100, grade C	3 blocked	unidentified
802	welt.de	Score 50 out of 100, grade D	3 blocked	Varnish
809	disneyplus.com	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
811	people.com	Score 37 out of 100, grade F, partial assessment	8 blocked	Cloudflare
817	adform.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
836	academia.edu	Score 48 out of 100, grade D, partial assessment	6 blocked	unidentified
837	lefigaro.fr	Score 55 out of 100, grade D	7 blocked	unidentified
843	daum.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
853	theverge.com	Score 44 out of 100, grade D	9 blocked	WordPress . Fastly
963	stbid.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
965	blackhub.team	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
969	mailchi.mp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
982	investopedia.com	Score 37 out of 100, grade F, partial assessment	8 blocked	Cloudflare
987	btloader.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1065	redfin.com	Score 90 out of 100, grade A, partial assessment	all allowed	Amazon CloudFront
1069	ouest-france.fr	Score 77 out of 100, grade B, partial assessment	1 blocked	unidentified
1071	rocketmoney.dev	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1083	bfmtv.com	Score 35 out of 100, grade F	11 blocked	Amazon CloudFront
1113	theconversation.com	Score 26 out of 100, grade F	11 blocked	Next.js . Fastly
1118	xiaohongshu.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1138	netcraze.io	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1196	fmkorea.com	Score 34 out of 100, grade F, partial assessment	6 blocked	unidentified
1228	namu.wiki	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1236	namecheaphosting.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1237	kakao.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1247	amap.com	Score 54 out of 100, grade D, partial assessment	5 blocked	unidentified
1254	furaffinity.netrefused GPTBot	Score 6 out of 100, grade F, partial assessment	11 blocked	Cloudflare
1287	themoviedb.org	Score 24 out of 100, grade F	11 blocked	Amazon CloudFront
1315	f95zone.torefused GPTBot	Score 27 out of 100, grade F	9 blocked	Cloudflare
1515	exhentai.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1522	attn.tv	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1548	tjk.org	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1571	aka.ms	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1663	sportfilm800.com	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1670	keirin.jp	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1691	twkan.comrefused GPTBot	Score 32 out of 100, grade F, partial assessment	8 blocked	Cloudflare
1752	bpexch.live	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1782	threads.net	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1790	qcloud.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1821	avito.st	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1823	newyorker.com	Score 51 out of 100, grade D	8 blocked	Amazon CloudFront
1858	zdnet.com	Score 44 out of 100, grade D	10 blocked	WordPress
1864	marketwatch.com	Score 31 out of 100, grade F, partial assessment	8 blocked	unidentified
1868	adrta.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1875	mega.nz	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
1905	wikihow.com	Score 31 out of 100, grade F	9 blocked	unidentified
1941	jamanetwork.comrefused GPTBot	Score 57 out of 100, grade D, partial assessment	5 blocked	Cloudflare
1960	sedoparking.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1976	secureserver.net	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
1982	clevertap-prod.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2008	vtb.ru	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2020	sogou.com	Score 6 out of 100, grade F, partial assessment	11 blocked	unidentified
2021	ign.comrefused GPTBot	Score 33 out of 100, grade F	10 blocked	unidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking FirecrawlAgent." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.