Top-level domain

.gov

Name: CrawlIndex dataset
License: https://creativecommons.org/licenses/by/4.0/

66 measured sites. 3 of them, 4.5%, block at least one answer-surface AI crawler. Mean readiness score 66.8.

Compare against every other top-level domain

Sites on .gov
Rank	Domain	Score	Answer-surface crawlers	Stack
125	nih.govrefused GPTBot	Score 83 out of 100, grade B, partial assessment	all allowed	Cloudflare
222	nist.gov	Score 68 out of 100, grade C	all allowed	Drupal . Cloudflare
340	cdc.gov	Score 100 out of 100, grade A, partial assessment	all allowed	Akamai
378	nasa.gov	Score 75 out of 100, grade B	all allowed	WordPress
575	noaa.gov	Not scored	all allowed	unidentified
581	ca.gov	Score 79 out of 100, grade B	all allowed	Azure Front Door
604	loc.govrefused GPTBot	Score 60 out of 100, grade C	all allowed	Cloudflare
751	fda.gov	Not scored	all allowed	unidentified
846	usda.gov	Not scored	all allowed	unidentified
911	state.gov	Score 59 out of 100, grade D	all allowed	WordPress . Amazon CloudFront
919	epa.gov	Score 62 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
986	whitehouse.gov	Score 83 out of 100, grade B	all allowed	WordPress
1321	irs.gov	Score 67 out of 100, grade C	all allowed	Drupal
1721	ftc.gov	Score 62 out of 100, grade C	all allowed	Drupal
1782	maricopa.govrefused GPTBot	Score 57 out of 100, grade D	all allowed	Cloudflare
1831	va.gov	Score 66 out of 100, grade C	all allowed	Next.js
1841	nps.gov	Score 64 out of 100, grade C	all allowed	Amazon CloudFront
2048	usgs.gov	Not scored	all allowed	unidentified
2092	ed.gov	Score 59 out of 100, grade D	all allowed	Drupal . Azure Front Door
2108	sec.gov	Score 100 out of 100, grade A, partial assessment	all allowed	Akamai
2123	bls.gov	Score 83 out of 100, grade B, partial assessment	all allowed	Akamai
2133	hhs.gov	Score 83 out of 100, grade B, partial assessment	all allowed	unidentified
2158	house.gov	Score 59 out of 100, grade D	all allowed	Drupal . Varnish
2193	justice.gov	Score 64 out of 100, grade C	all allowed	Drupal
2224	census.gov	Score 67 out of 100, grade C	all allowed	Adobe Experience Manager . Cloudflare
2361	energy.gov	Score 66 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
2367	senate.gov	Score 56 out of 100, grade D	all allowed	unidentified
2370	congress.govrefused GPTBot	Score 25 out of 100, grade F, partial assessment	10 blocked	Cloudflare
2386	weather.gov	Score 56 out of 100, grade D	all allowed	unidentified
2430	ny.gov	Score 100 out of 100, grade A, partial assessment	all allowed	Cloudflare
2549	ssa.gov	Not scored	all allowed	unidentified
2590	dhs.govrefused GPTBot	Score 57 out of 100, grade D	all allowed	Drupal . Akamai
2692	medlineplus.gov	Score 79 out of 100, grade B	all allowed	Amazon CloudFront
2740	archives.gov	Score 71 out of 100, grade C	all allowed	Amazon CloudFront
2803	fcc.govrefused GPTBot	Score 83 out of 100, grade B, partial assessment	all allowed	Akamai
2814	dot.gov	Score 83 out of 100, grade B, partial assessment	all allowed	Akamai
2822	treasury.gov	Score 59 out of 100, grade D	all allowed	Drupal . Akamai
2830	privacyshield.gov	Not scored	all allowed	unidentified
2846	wa.gov	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
2889	texas.gov	Score 76 out of 100, grade B	all allowed	Gatsby . Amazon CloudFront
2933	usa.gov	Score 79 out of 100, grade B	all allowed	Drupal . Amazon CloudFront
2970	fbi.govrefused GPTBot	Score 83 out of 100, grade B, partial assessment	all allowed	Cloudflare
3049	uspto.gov	Score 83 out of 100, grade B	all allowed	Drupal . Amazon CloudFront
3089	dol.gov	Score 89 out of 100, grade B, partial assessment	all allowed	Akamai
3103	cancer.gov	Score 68 out of 100, grade C	all allowed	Drupal
3168	nyc.gov	Score 62 out of 100, grade C	all allowed	Adobe Experience Manager . Fastly
3236	usembassy.gov	Not scored	all allowed	unidentified
3401	federalregister.govrefused GPTBot	Score 57 out of 100, grade D	all allowed	unidentified
3579	dataprivacyframework.gov	Score 48 out of 100, grade D	all allowed	unidentified
3624	sba.gov	Score 83 out of 100, grade B	all allowed	WordPress . Amazon CloudFront
3632	clinicaltrials.gov	Score 57 out of 100, grade D	all allowed	Google Cloud / GFE
3641	osha.gov	Score 59 out of 100, grade D	all allowed	Drupal . Amazon CloudFront
3721	cia.gov	Score 64 out of 100, grade C	all allowed	Gatsby . Akamai
3795	uscourts.gov	Score 62 out of 100, grade C	all allowed	Drupal . Akamai
3824	uscis.govrefused GPTBot	Score 52 out of 100, grade D	all allowed	Drupal
3840	cisa.govrefused GPTBot	Score 52 out of 100, grade D	all allowed	Drupal
3860	fema.govrefused GPTBot	Score 83 out of 100, grade B, partial assessment	1 blocked	Akamai
3963	federalreserve.gov	Score 59 out of 100, grade D	all allowed	Cloudflare
4114	cms.gov	Score 67 out of 100, grade C	all allowed	Drupal . Akamai
4168	virginia.gov	Score 61 out of 100, grade C	all allowed	unidentified
4170	faa.gov	Score 56 out of 100, grade D	all allowed	Drupal
4206	bldrdoc.gov	Not scored	all allowed	unidentified
4211	eia.gov	Score 67 out of 100, grade C	all allowed	unidentified
4233	studentaid.gov	Score 45 out of 100, grade D	all allowed	Drupal . Akamai
4237	defense.gov	Score 83 out of 100, grade B, partial assessment	all allowed	Akamai
4561	nsf.gov	Score 67 out of 100, grade C	all allowed	Drupal . Amazon CloudFront
4639	mo.gov	Score 62 out of 100, grade C	all allowed	Drupal . Imperva
4749	pa.gov	Score 67 out of 100, grade C	all allowed	Adobe Experience Manager . Fastly
4836	ct.gov	Score 59 out of 100, grade D	all allowed	Sitecore . Azure Front Door
4853	govinfo.gov	Score 64 out of 100, grade C	all allowed	Drupal . Cloudflare
5012	in.gov	Score 48 out of 100, grade D	all allowed	unidentified
5085	cbp.govrefused GPTBot	Score 55 out of 100, grade D	all allowed	Drupal
5123	mass.gov	Score 83 out of 100, grade B, partial assessment	all allowed	unidentified
5126	nj.gov	Score 59 out of 100, grade D	all allowed	Imperva

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Agent readiness on .gov." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.