crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Leaderboard

Who is ready for agents, and who is not

Ranked by the CrawlIndex Score across 2,613 fully measured sites. One row per operator, because a company publishing the same policy on twenty country domains is one fact rather than twenty.

Number of measured sites in each ten-point score band
Score bandSites
0 to 90
10 to 198
20 to 2918
30 to 3953
40 to 49293
50 to 59669
60 to 69751
70 to 79477
80 to 89321
90 to 9923

521 sites score 75 or above and qualify for the Agent Friendly mark or better. The rest of the distribution sits in the middle: not hostile to agents, just never set up for them.

Only fully measured sites appear here.

Partial assessmentsSome checks could not be observed, usually because a bot wall answered instead of the site, so those points were removed from the total rather than failed. The remaining points are renormalised to one hundred. A partial score is not comparable with a complete one, which is why partial sites are kept out of ranked lists. More
are renormalised over fewer points, so putting one in the same column would compare two different scales. Sites served a
stub responseA homepage that answered with under 5,000 bytes and no readable text. That is an anti-automation placeholder served to our crawler, not the site. Everything derived from it is discarded rather than counted against the operator. More
are partial by definition, which is why some very large names are absent.

Looking for one domain rather than a ranking? Press / or use search.

Most agent-ready

Open to the crawlers that answer questions, publishing something machine-readable, and serving content a crawler can read without running JavaScript.

Highest scoring operators
RankDomainScorePolicyStack
438appsflyer.comScore 95 out of 100, grade AOpenInheritedWordPress . Cloudflare
--crawlindex.orgScore 95 out of 100, grade AOpenInheritedNext.js . Vercel
1316oxylabs.ioScore 93 out of 100, grade AOpenDeliberateNext.js . Cloudflare
1564therapservices.netScore 93 out of 100, grade AOpenDeliberateWordPress . Cloudflare
2374wpengine.comScore 93 out of 100, grade AOpenInheritedWordPress . Cloudflare
2167railway.appScore 92 out of 100, grade AOpenInheritedunidentified
570atlassian.com
and 1 more domain with the same policy
Score 91 out of 100, grade AOpenInheritedAmazon CloudFront
1108usercentrics.euScore 91 out of 100, grade AOpenInheritedWordPress . Cloudflare
1328haberler.comScore 91 out of 100, grade AOpenDeliberateCloudflare
1411protothema.grScore 91 out of 100, grade AOpenInheritedunidentified
1486sondakika.comScore 91 out of 100, grade AOpenDeliberateCloudflare
2113yotpo.comScore 91 out of 100, grade AOpenDeliberateWordPress . Amazon CloudFront
2672duosecurity.comScore 91 out of 100, grade AOpenInheritedAmazon CloudFront
2682dnsimple.comScore 91 out of 100, grade AOpenInheritedunidentified
2813woocommerce.comScore 91 out of 100, grade AOpenInheritedWooCommerce
2866mountain.comScore 91 out of 100, grade AOpenInheritedWordPress . Cloudflare
2943crazyegg.comScore 91 out of 100, grade AOpenDeliberateCloudflare
3168elevenlabs.ioScore 91 out of 100, grade AOpenInheritedNext.js . Google Cloud / GFE
4489helpshift.comScore 91 out of 100, grade AOpenInheritedHubSpot CMS . Cloudflare
4575adspower.netScore 91 out of 100, grade AOpenInheritedAmazon CloudFront
1852sendbird.comScore 90 out of 100, grade AOpenDeliberateNuxt . Netlify
2927redis.ioScore 90 out of 100, grade AOpenDeliberateSanity . Cloudflare
3770colorlib.comScore 90 out of 100, grade AOpenInheritedWordPress . Cloudflare
68wordpress.org
and 1 more domain with the same policy
Score 89 out of 100, grade BOpenDeliberateWordPress
360branch.ioScore 89 out of 100, grade BOpenDeliberateWordPress . Cloudflare
361dynatrace.comScore 89 out of 100, grade BOpenInheritedNuxt . Amazon CloudFront
503foxnews.comScore 89 out of 100, grade BOpenInheritedFastly
523klaviyo.comScore 89 out of 100, grade BOpenDeliberateHubSpot CMS . Cloudflare
633stackadapt.comScore 89 out of 100, grade BOpenInheritedWordPress
794cloudinary.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
1266singular.netScore 89 out of 100, grade BOpenDeliberateWordPress . Cloudflare
1537musicdel.irScore 89 out of 100, grade BOpenInheritedWordPress
1897hostinger.comScore 89 out of 100, grade BOpenDeliberateNuxt . Cloudflare
2104datto.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
2256iubenda.comScore 89 out of 100, grade BOpenInheritedWordPress . Bunny
2401brightspace.comScore 89 out of 100, grade BOpenDeliberateWordPress . Cloudflare
2505urbanairship.comScore 89 out of 100, grade BOpenInheritedWordPress . Fastly
2569pubnative.netScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
2576sardine.aiScore 89 out of 100, grade BOpenInheritedSanity . Vercel
2728sportradar.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
2868cloudx.ioScore 89 out of 100, grade BOpenInheritedNext.js . Vercel
2876cutt.lyScore 89 out of 100, grade BOpenInheritedCloudflare
2879edna.ruScore 89 out of 100, grade BOpenInheritedWordPress
3178iterable.comScore 89 out of 100, grade BOpenInheritedWordPress . Fastly
3224ably.io
and 1 more domain with the same policy
Score 89 out of 100, grade BOpenInheritedGatsby . Cloudflare
3479demandbase.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
3544visualwebsiteoptimizer.comScore 89 out of 100, grade BOpenDeliberateWordPress . Cloudflare
3806signifyd.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
3930tum.deScore 89 out of 100, grade BOpenInheritedTYPO3
4115thescore.comScore 89 out of 100, grade BOpenDeliberateNext.js . Cloudflare
4268termly.ioScore 89 out of 100, grade BOpenDeliberateHubSpot CMS . Cloudflare
4552salesmartly.comScore 89 out of 100, grade BOpenInheritedNext.js . Amazon CloudFront
4608popsci.comScore 89 out of 100, grade BOpenInheritedWordPress
4625niceincontact.comScore 89 out of 100, grade BOpenInheritedWordPress
4663cyberfolks.plScore 89 out of 100, grade BOpenInheritedWordPress
4665controlup.comScore 89 out of 100, grade BOpenInheritedWordPress
4972magnite.comScore 89 out of 100, grade BOpenInheritedWordPress . Cloudflare
254centinel-analytica.comScore 88 out of 100, grade BOpenDeliberateNext.js . Cloudflare
621intercom.io
and 1 more domain with the same policy
Score 88 out of 100, grade BOpenInheritedSanity . Amazon CloudFront
764auth0.comScore 88 out of 100, grade BOpenInheritedNext.js . Cloudflare
1845pewresearch.orgScore 88 out of 100, grade BOpenInheritedWordPress
1901utorrent.comScore 88 out of 100, grade BOpenDeliberateGatsby . Amazon CloudFront
1932wpastra.comScore 88 out of 100, grade BOpenInheritedHubSpot CMS . Cloudflare
2071livechatinc.comScore 88 out of 100, grade BOpenInheritedHubSpot CMS . Akamai
2179acronis.comScore 88 out of 100, grade BOpenInheritedWordPress . Cloudflare
2338mindbox.ruScore 88 out of 100, grade BOpenDeliberateunidentified
2989clerk.comScore 88 out of 100, grade BOpenInheritedNext.js . Vercel
2991zapier.comScore 88 out of 100, grade BOpenDeliberateContentful . Amazon CloudFront
3029elespanol.comScore 88 out of 100, grade BOpenInheritedunidentified
3046intercomcdn.comScore 88 out of 100, grade BOpenInheritedSanity . Amazon CloudFront
3144walkme.comScore 88 out of 100, grade BOpenInheritedWordPress
3355amplify.comScore 88 out of 100, grade BOpenDeliberateHubSpot CMS . Amazon CloudFront
3739tanium.comScore 88 out of 100, grade BOpenDeliberateSanity . Cloudflare
3751netlify.com
and 1 more domain with the same policy
Score 88 out of 100, grade BOpenDeliberateHubSpot CMS . Netlify
3857domaintools.comScore 88 out of 100, grade BOpenInheritedCloudflare
4215picsart.comScore 88 out of 100, grade BOpenInheritedNext.js . Cloudflare
4716livechat.comScore 88 out of 100, grade BOpenInheritedHubSpot CMS . Akamai
--fidgetlabs.io
and 1 more domain with the same policy
Score 88 out of 100, grade BOpenDeliberateNext.js . Vercel
183forter.comScore 87 out of 100, grade BNo trainingDeliberateWordPress . Cloudflare
1836moengage.comScore 87 out of 100, grade BOpenInheritedWordPress . Cloudflare
2051customer.ioScore 87 out of 100, grade BOpenInheritedSanity . Vercel
2479gameanalytics.comScore 87 out of 100, grade BOpenDeliberateCloudflare
2762newsbreak.comScore 87 out of 100, grade BOpenDeliberateNext.js . Amazon CloudFront
2976vercel.comScore 87 out of 100, grade BOpenInheritedNext.js . Vercel
3344adsbynimbus.comScore 87 out of 100, grade BOpenInheritedCloudflare
3608nanit.comScore 87 out of 100, grade BOpenDeliberateShopify . Cloudflare
3775openrouter.aiScore 87 out of 100, grade BOpenInheritedNext.js . Cloudflare
4023complianz.ioScore 87 out of 100, grade BOpenInheritedHubSpot CMS . Cloudflare
4422parentsquare.comScore 87 out of 100, grade BOpenInheritedWordPress
4476zetaglobal.netScore 87 out of 100, grade BOpenDeliberateNext.js . Vercel
123samsung.comScore 86 out of 100, grade BOpenDeliberateAdobe Experience Manager . Akamai
196kaspersky.comScore 86 out of 100, grade BOpenDeliberateNext.js
316zendesk.comScore 86 out of 100, grade BOpenInheritedNext.js . Cloudflare
628onetrust.comScore 86 out of 100, grade BOpenInheritedAdobe Experience Manager . Cloudflare
718gumgum.comScore 86 out of 100, grade BOpenDeliberateCloudflare
871unpkg.comScore 86 out of 100, grade BOpenDeliberateCloudflare
2107crowdstrike.comScore 86 out of 100, grade BOpenInheritedAdobe Experience Manager . Fastly
2175primis.techScore 86 out of 100, grade BOpenInheritedHubSpot CMS . Amazon CloudFront
2211nielsen.comScore 86 out of 100, grade BOpenInheritedWordPress
2240useinsider.comScore 86 out of 100, grade BOpenInheritedHubSpot CMS . Cloudflare

Least agent-ready

A low score here is not an accusation of bad faith. Most of these sites blocked nothing deliberately. Someone else usually decided.

Lowest scoring operators
RankDomainScorePolicyStack
63tiktok.comScore 15 out of 100, grade FWalledDeliberateunidentified
632amazon.fr
and 5 more domains with the same policy
Score 15 out of 100, grade FSelectiveDeliberateAmazon CloudFront
3981findagrave.comScore 19 out of 100, grade FWalledDeliberateCloudflare
296launchpad.netScore 20 out of 100, grade FSelectiveDeliberateunidentified
4601gentoo.orgScore 21 out of 100, grade FWalledDeliberateFastly
486cnbc.comScore 24 out of 100, grade FWalledDeliberateAkamai
1402dw.comScore 24 out of 100, grade FSelectiveDeliberateAkamai
1612wnacg.comScore 24 out of 100, grade FAssistant onlyDeliberateCloudflare
3574politico.euScore 24 out of 100, grade FWalledDeliberateWordPress . Cloudflare
1195theconversation.comScore 26 out of 100, grade FWalledDeliberateNext.js . Fastly
1407iltalehti.fiScore 26 out of 100, grade FWalledDeliberateAmazon CloudFront
2772faz.netScore 26 out of 100, grade FSelectiveDeliberateNuxt
3273franceinfo.frScore 27 out of 100, grade FWalledDeliberateunidentified
4744stuff.co.nzScore 27 out of 100, grade FSelectiveDeliberateFastly
1447ekstrabladet.dkScore 28 out of 100, grade FWalledDeliberateFastly
1753bpexch.live
and 1 more domain with the same policy
Score 28 out of 100, grade FNo trainingDeliberateCloudflare
4085nrk.noScore 28 out of 100, grade FSelectiveDeliberateSvelteKit
1050pimpbunny.comScore 29 out of 100, grade FSelectiveDeliberateunidentified
1479joyclub.deScore 29 out of 100, grade FWalledDeliberateCloudflare
2148ign.comScore 29 out of 100, grade FSelectiveDeliberateWordPress . Fastly
882nbcnews.comScore 30 out of 100, grade FWalledDeliberateNext.js . Akamai
2021videeyss.shopScore 30 out of 100, grade FNo trainingDeliberateCloudflare
4823band.usScore 30 out of 100, grade FAssistant onlyDeliberateunidentified
192cnn.comScore 31 out of 100, grade FSelectiveDeliberateFastly
219bbc.comScore 31 out of 100, grade FSelectiveDeliberateNext.js . Fastly
1361dagbladet.noScore 31 out of 100, grade FWalledDeliberateCloudflare
1977mercadolibre.com
and 2 more domains with the same policy
Score 31 out of 100, grade FSelectiveDeliberateunidentified
4238channel4.comScore 31 out of 100, grade FSelectiveDeliberateAkamai
4798dr.dkScore 31 out of 100, grade FWalledDeliberateNext.js
4875yr.noScore 31 out of 100, grade FSelectiveDeliberateunidentified
5013windstream.netScore 31 out of 100, grade FSelectiveDeliberateVarnish
1225gazzetta.itScore 32 out of 100, grade FWalledDeliberateunidentified
2667mastodon.socialScore 32 out of 100, grade FNo trainingDeliberateFastly
3228metro.co.ukScore 32 out of 100, grade FWalledDeliberateWordPress
3401today.comScore 32 out of 100, grade FWalledDeliberateNext.js . Akamai
1645dogdrip.netScore 33 out of 100, grade FNo trainingDeliberateCloudflare
1789blacktoon423.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2086filmyzilla68.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2091blacktoon421.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2093filmyzilla65.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
2094filmyzilla67.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
39652osb.comScore 33 out of 100, grade FNo trainingDeliberateCloudflare
242bbc.co.ukScore 34 out of 100, grade FSelectiveDeliberateFastly
955corriere.itScore 34 out of 100, grade FWalledDeliberateFastly
2030sportfilm800.comScore 34 out of 100, grade FNo trainingDeliberateNext.js . Cloudflare
4009thestar.comScore 34 out of 100, grade FWalledDeliberateunidentified
582dailymail.com
and 1 more domain with the same policy
Score 35 out of 100, grade FWalledDeliberateAkamai
1063bfmtv.comScore 35 out of 100, grade FWalledDeliberateAmazon CloudFront
2216mashable.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
2558pcmag.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
2959asahi.comScore 35 out of 100, grade FWalledDeliberateAkamai
3444lifehacker.comScore 35 out of 100, grade FSelectiveDeliberateCloudflare
427520minutes.frScore 35 out of 100, grade FSelectiveDeliberateCloudflare
4595p4455.comScore 35 out of 100, grade FNo trainingDeliberateWordPress . Cloudflare
4681test-uz.ruScore 35 out of 100, grade FSelectiveDeliberateunidentified
1292actu.frScore 36 out of 100, grade FSelectiveDeliberateCloudflare
1508pussyspace.comScore 36 out of 100, grade FNo trainingDeliberateCloudflare
1767hdhub4u.biScore 36 out of 100, grade FNo trainingDeliberateCloudflare
2078moviezwap.bestScore 36 out of 100, grade FNo trainingDeliberateCloudflare
2921chicagotribune.comScore 36 out of 100, grade FWalledDeliberateWordPress
3763nydailynews.comScore 36 out of 100, grade FWalledDeliberateWordPress
630telegraph.co.ukScore 37 out of 100, grade FSelectiveDeliberateAdobe Experience Manager . Fastly
3358toyota.comScore 37 out of 100, grade FAssistant onlyDeliberateAdobe Experience Manager . Amazon CloudFront
914hbr.orgScore 38 out of 100, grade FNo trainingDeliberateNext.js . Amazon CloudFront
1539shahvani.comScore 38 out of 100, grade FNo trainingDeliberateCloudflare
1817buzzfeed.comScore 38 out of 100, grade FSelectiveDeliberateNext.js . Fastly
2068filmyfly.bingoScore 38 out of 100, grade FNo trainingDeliberateCloudflare
1501xnxx.healthScore 39 out of 100, grade FNo trainingDeliberateCloudflare
2579elmundo.esScore 39 out of 100, grade FWalledDeliberateFastly
3338orf.atScore 39 out of 100, grade FNo trainingDeliberateunidentified
4005radiofrance.frScore 39 out of 100, grade FSelectiveDeliberateSvelteKit . Akamai
4322kompas.comScore 39 out of 100, grade FSelectiveDeliberateAmazon CloudFront
81amazonvideo.comScore 40 out of 100, grade DSelectiveDeliberateAmazon CloudFront
690spiegel.deScore 40 out of 100, grade DSelectiveDeliberateGoogle Cloud / GFE
1043pandora.comScore 40 out of 100, grade DNo trainingDeliberateunidentified
2714ko-fi.comScore 40 out of 100, grade DNo trainingDeliberateCloudflare
3839ilsole24ore.comScore 40 out of 100, grade DSelectiveDeliberateVarnish
3859yomiuri.co.jpScore 40 out of 100, grade DWalledDeliberateWordPress . Fastly
751shopee.co.idpolicy gap
and 7 more domains with the same policy
Score 41 out of 100, grade DOpenInheritedunidentified
1109cookpad.comScore 41 out of 100, grade DSelectiveDeliberateFastly
1815expressen.seScore 41 out of 100, grade DSelectiveDeliberateunidentified
3815oregonstate.eduScore 41 out of 100, grade DNo trainingDeliberateDrupal . Cloudflare
3957uiowa.eduScore 41 out of 100, grade DNo trainingDeliberateDrupal . Cloudflare
4018derstandard.atScore 41 out of 100, grade DSelectiveDeliberateunidentified
293withgoogle.comScore 42 out of 100, grade DUndeclaredAbsentGoogle Cloud / GFE
905mercadolivre.com.brScore 42 out of 100, grade DSelectiveDeliberateunidentified
1093marca.comScore 42 out of 100, grade DWalledDeliberateFastly
1301zhihu.compolicy gapScore 42 out of 100, grade DUndeclaredAbsentunidentified
1429meb.gov.trScore 42 out of 100, grade DSelectiveDeliberateunidentified
1432eenadu.netScore 42 out of 100, grade DWalledDeliberateAmazon CloudFront
4691foreignpolicy.comScore 42 out of 100, grade DSelectiveDeliberateWordPress . Cloudflare
72yahoo.com
and 1 more domain with the same policy
Score 43 out of 100, grade DSelectiveDeliberateNext.js
509statista.comScore 43 out of 100, grade DSelectiveDeliberateAmazon CloudFront
1682arabx.camScore 43 out of 100, grade DNo trainingDeliberateCloudflare
2001iqiyi.comScore 43 out of 100, grade DSelectiveDeliberateunidentified
2816abc.esScore 43 out of 100, grade DSelectiveDeliberateWordPress
4260crossthegame.compolicy gapScore 43 out of 100, grade DOpenInheritedCloudflare
756cnet.comScore 44 out of 100, grade DSelectiveDeliberateWordPress
853theverge.comScore 44 out of 100, grade DSelectiveDeliberateWordPress . Fastly
1885zdnet.comScore 44 out of 100, grade DSelectiveDeliberateWordPress

crawl 2026-11-22 10:38 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,656 of 5,006 reachable

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Agent readiness leaderboard." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.