Top-level domain

.gov

Name: CrawlIndex dataset
License: https://creativecommons.org/licenses/by/4.0/

66 measured sites. 3 of them, 4.5%, block at least one answer-surface AI crawler. Mean readiness score 65.6.

Compare against every other top-level domain

Sites on .gov
Rank	Domain	Score	Answer-surface crawlers	Stack
143	nih.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	all allowed	Cloudflare
237	nist.gov	Score 67 out of 100, grade C	all allowed	Drupal . Cloudflare
361	cdc.gov	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
395	nasa.gov	Score 75 out of 100, grade B	all allowed	WordPress
531	noaa.govrefused GPTBot	Score 50 out of 100, grade D	all allowed	Drupal . Amazon CloudFront
599	ca.gov	Score 76 out of 100, grade B	all allowed	Azure Front Door
652	loc.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	all allowed	Cloudflare
847	usda.gov	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
936	epa.gov	Score 63 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
952	state.govrefused GPTBot	Score 53 out of 100, grade D	all allowed	WordPress . Amazon CloudFront
1036	whitehouse.gov	Score 80 out of 100, grade B	all allowed	WordPress
1067	maricopa.govrefused GPTBot	Score 54 out of 100, grade D	all allowed	Cloudflare
1367	irs.gov	Score 61 out of 100, grade C	all allowed	Drupal . Akamai
1690	ftc.gov	Score 63 out of 100, grade C	all allowed	Drupal
1792	va.gov	Score 63 out of 100, grade C	all allowed	Next.js
1856	nps.gov	Score 61 out of 100, grade C	all allowed	Amazon CloudFront
1981	sec.gov	Score 92 out of 100, grade A, partial assessment	all allowed	Akamai
1991	ed.gov	Score 60 out of 100, grade C	all allowed	Drupal . Azure Front Door
2041	bls.gov	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
2069	hhs.gov	Score 78 out of 100, grade B, partial assessment	all allowed	unidentified
2109	house.gov	Score 57 out of 100, grade D	all allowed	Drupal . Varnish
2130	census.gov	Score 69 out of 100, grade C	all allowed	Adobe Experience Manager . Cloudflare
2158	justice.gov	Score 64 out of 100, grade C	all allowed	Drupal
2226	weather.gov	Score 55 out of 100, grade D	all allowed	unidentified
2275	energy.gov	Score 66 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
2303	congress.govrefused GPTBot	Score 21 out of 100, grade F, partial assessment	10 blocked	Cloudflare
2381	ny.gov	Score 92 out of 100, grade A, partial assessment	all allowed	Cloudflare
2382	senate.gov	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
2586	dhs.govrefused GPTBot	Score 57 out of 100, grade D	all allowed	Drupal . Akamai
2709	archives.gov	Score 67 out of 100, grade C	all allowed	Amazon CloudFront
2710	medlineplus.gov	Score 74 out of 100, grade C	all allowed	Amazon CloudFront
2743	faa.gov	Score 57 out of 100, grade D	all allowed	Drupal
2832	texas.gov	Score 71 out of 100, grade C	all allowed	Gatsby . Amazon CloudFront
2875	treasury.gov	Score 57 out of 100, grade D	all allowed	Akamai
2887	dot.gov	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
2890	fcc.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
2929	wa.gov	Score 14 out of 100, grade F, partial assessment	11 blocked	unidentified
3040	usa.gov	Score 74 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
3153	uspto.gov	Score 80 out of 100, grade B	all allowed	Drupal . Amazon CloudFront
3171	fbi.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	all allowed	Cloudflare
3214	cancer.gov	Score 64 out of 100, grade C	all allowed	Drupal
3219	dol.govrefused GPTBot	Score 84 out of 100, grade B, partial assessment	all allowed	Akamai
3259	nyc.gov	Score 63 out of 100, grade C	all allowed	Adobe Experience Manager . Fastly
3332	federalregister.govrefused GPTBot	Score 55 out of 100, grade D	all allowed	unidentified
3502	dataprivacyframework.gov	Score 84 out of 100, grade B, partial assessment	all allowed	unidentified
3695	osha.govrefused GPTBot	Score 47 out of 100, grade D	all allowed	Drupal . Amazon CloudFront
3707	clinicaltrials.gov	Score 56 out of 100, grade D	all allowed	Google Cloud / GFE
3737	sba.gov	Score 80 out of 100, grade B	all allowed	WordPress . Amazon CloudFront
3763	cisa.govrefused GPTBot	Score 53 out of 100, grade D	all allowed	Drupal
3944	uscourts.gov	Score 63 out of 100, grade C	all allowed	Drupal . Akamai
3963	cia.gov	Score 61 out of 100, grade C	all allowed	Gatsby . Akamai
3988	federalreserve.gov	Score 57 out of 100, grade D	all allowed	Cloudflare
4015	uscis.govrefused GPTBot	Score 53 out of 100, grade D	all allowed	Drupal
4024	fema.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	1 blocked	Akamai
4149	eia.gov	Score 64 out of 100, grade C	all allowed	unidentified
4159	virginia.gov	Score 59 out of 100, grade D	all allowed	unidentified
4231	cms.gov	Score 64 out of 100, grade C	all allowed	Drupal . Akamai
4313	studentaid.gov	Score 45 out of 100, grade D	all allowed	Drupal . Akamai
4543	defense.govrefused GPTBot	Score 78 out of 100, grade B, partial assessment	all allowed	Akamai
4676	nsf.gov	Score 67 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
4719	mo.gov	Score 60 out of 100, grade C	all allowed	Drupal . Imperva
4729	ct.gov	Score 57 out of 100, grade D	all allowed	Sitecore . Azure Front Door
4744	pa.gov	Score 64 out of 100, grade C	all allowed	Adobe Experience Manager . Fastly
4814	in.gov	Score 92 out of 100, grade A, partial assessment	all allowed	unidentified
5073	nj.gov	Score 57 out of 100, grade D	all allowed	Imperva
5105	cbp.govrefused GPTBot	Score 53 out of 100, grade D	all allowed	Drupal

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Agent readiness on .gov." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.