crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Score band . grade F

Sites scoring 30 to 39

53 of 2,613 fully measured sites land here, 2.0% of the index. Blocks most answer-surface crawlers, or serves almost nothing a crawler can read.

52 operators

One row per operator, highest score first. Partial assessments are excluded, because a renormalised score belongs to a different scale and could not be placed in a band honestly.

Sites scoring 30 to 39
RankDomainScorePolicyStack
2579elmundo.esScore 39 out of 100, grade FWalledDeliberateFastly
4322kompas.comScore 39 out of 100, grade FSelectiveDeliberateAmazon CloudFront
3338orf.atScore 39 out of 100, grade FNo trainingDeliberateunidentified
4005radiofrance.frScore 39 out of 100, grade FSelectiveDeliberateSvelteKit . Akamai
1501xnxx.healthScore 39 out of 100, grade FNo trainingDeliberateCloudflare
1817buzzfeed.comScore 38 out of 100, grade FSelectiveDeliberateNext.js . Fastly
2068filmyfly.bingoScore 38 out of 100, grade FNo trainingDeliberateCloudflare
914hbr.orgScore 38 out of 100, grade FNo trainingDeliberateNext.js . Amazon CloudFront
1539shahvani.comScore 38 out of 100, grade FNo trainingDeliberateCloudflare
630telegraph.co.ukScore 37 out of 100, grade FSelectiveDeliberateAdobe Experience Manager . Fastly
3358toyota.comScore 37 out of 100, grade FAssistant onlyDeliberateAdobe Experience Manager . Amazon CloudFront
1292actu.frScore 36 out of 100, grade FSelectiveDeliberateCloudflare
2921chicagotribune.comScore 36 out of 100, grade FWalledDeliberateWordPress
1767hdhub4u.biScore 36 out of 100, grade FNo trainingDeliberateCloudflare
2078moviezwap.bestScore 36 out of 100, grade FNo trainingDeliberateCloudflare
3763nydailynews.comScore 36 out of 100, grade FWalledDeliberateWordPress
1508pussyspace.comScore 36 out of 100, grade FNo trainingDeliberateCloudflare
427520minutes.frScore 35 out of 100, grade FSelectiveDeliberateCloudflare
2959asahi.comScore 35 out of 100, grade FWalledDeliberateAkamai
1063bfmtv.comScore 35 out of 100, grade FWalledDeliberateAmazon CloudFront
734dailymail.co.uk
and 1 more domain with the same policy
Score 35 out of 100, grade FWalledDeliberateAkamai
3444lifehacker.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
2216mashable.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
4595p4455.comScore 35 out of 100, grade FNo trainingDeliberateWordPress . Cloudflare
2558pcmag.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
4681test-uz.ruScore 35 out of 100, grade FSelectiveDeliberateunidentified
242bbc.co.ukScore 34 out of 100, grade FSelectiveDeliberateFastly
955corriere.itScore 34 out of 100, grade FWalledDeliberateFastly
2030sportfilm800.comScore 34 out of 100, grade FNo trainingDeliberateNext.js . Cloudflare
4009thestar.comScore 34 out of 100, grade FWalledDeliberateunidentified
39652osb.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2091blacktoon421.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
1789blacktoon423.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
1645dogdrip.netScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2093filmyzilla65.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2094filmyzilla67.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2086filmyzilla68.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
1225gazzetta.itScore 32 out of 100, grade FWalledDeliberateunidentified
2667mastodon.socialScore 32 out of 100, grade FNo trainingDeliberateFastly
3228metro.co.ukScore 32 out of 100, grade FWalledDeliberateWordPress
3401today.comScore 32 out of 100, grade FWalledDeliberateNext.js . Akamai
219bbc.comScore 31 out of 100, grade FSelectiveDeliberateNext.js . Fastly
4238channel4.comScore 31 out of 100, grade FSelectiveDeliberateAkamai
192cnn.comScore 31 out of 100, grade FSelectiveDeliberateFastly
1361dagbladet.noScore 31 out of 100, grade FWalledDeliberateCloudflare
4798dr.dkScore 31 out of 100, grade FWalledDeliberateNext.js
1977mercadolibre.comScore 31 out of 100, grade FSelectiveDeliberateunidentified
5013windstream.netScore 31 out of 100, grade FSelectiveDeliberateVarnish
4875yr.noScore 31 out of 100, grade FSelectiveDeliberateunidentified
4823band.usScore 30 out of 100, grade FAssistant onlyDeliberateunidentified
882nbcnews.comScore 30 out of 100, grade FWalledDeliberateNext.js . Akamai
2021videeyss.shopScore 30 out of 100, grade FNo trainingDeliberateCloudflare

Showing the first 52 operators. The full list is in the dataset.

crawl 2026-11-22 10:38 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,656 of 5,006 reachable

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites scoring 30 to 39 for AI readiness." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.