Top-level domain
.com
1,970 measured sites. 361 of them, 18.3%, block at least one answer-surface AI crawler. Mean readiness score 66.
Compare against every other top-level domain
| Rank | Domain | Score | Answer-surface crawlers | Stack |
|---|---|---|---|---|
| 1 | google.comrefused GPTBot | Score 49 out of 100, grade D | all allowed | Google Cloud / GFE |
| 2 | cloudflare.com | Score 85 out of 100, grade B | all allowed | Astro . Cloudflare |
| 5 | facebook.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 9 | microsoft.com | Score 65 out of 100, grade C | all allowed | Adobe Experience Manager . Azure Front Door |
| 10 | youtube.com | Score 56 out of 100, grade D | all allowed | Google Cloud / GFE |
| 12 | apple.com | Score 77 out of 100, grade B | all allowed | unidentified |
| 14 | instagram.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 16 | twitter.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 17 | linkedin.com | Score 17 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 19 | triplinkintl.com | Score 57 out of 100, grade D | all allowed | unidentified |
| 24 | appsflyersdk.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | Amazon CloudFront |
| 29 | amazon.comrefused GPTBot | Score 16 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 31 | sharepoint.com | Score 55 out of 100, grade D | all allowed | Adobe Experience Manager . Azure Front Door |
| 34 | github.com | Score 72 out of 100, grade C | all allowed | Contentful |
| 38 | lsrelayaccess.com | Score 78 out of 100, grade B, partial assessment | all allowed | unidentified |
| 43 | netflix.com | Score 59 out of 100, grade D, partial assessment | 3 blocked | unidentified |
| 46 | bing.com | Score 61 out of 100, grade C | all allowed | unidentified |
| 48 | x.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 55 | pinterest.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 56 | icloud.com | Score 52 out of 100, grade D | all allowed | unidentified |
| 61 | adobe.com | Score 92 out of 100, grade A, partial assessment | all allowed | Akamai |
| 63 | tiktok.com | Score 15 out of 100, grade F | 11 blocked | unidentified |
| 70 | googledomains.com | Score 52 out of 100, grade D | all allowed | unidentified |
| 72 | yahoo.comrefused GPTBot | Score 43 out of 100, grade D | 5 blocked | Next.js |
| 74 | digicert.com | Score 92 out of 100, grade A, partial assessment | all allowed | Imperva |
| 78 | spotify.com | Score 52 out of 100, grade D | all allowed | Fastly |
| 79 | opera.com | Score 85 out of 100, grade B | all allowed | unidentified |
| 80 | epicgames.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 81 | amazonvideo.com | Score 40 out of 100, grade D | 5 blocked | Amazon CloudFront |
| 82 | roblox.com | Score 61 out of 100, grade C | all allowed | unidentified |
| 84 | vimeo.com | Score 56 out of 100, grade D | 3 blocked | Next.js . Cloudflare |
| 90 | baidu.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 91 | qq.com | Score 55 out of 100, grade D | all allowed | unidentified |
| 92 | blogspot.com | Score 57 out of 100, grade D | all allowed | unidentified |
| 98 | intuit.com | Score 92 out of 100, grade A, partial assessment | all allowed | Akamai |
| 99 | f5.com | Score 72 out of 100, grade C | all allowed | Sanity . Vercel |
| 100 | wordpress.comrefused GPTBot | Score 77 out of 100, grade B | all allowed | WordPress |
| 104 | nginx.com | Score 61 out of 100, grade C | all allowed | Next.js . Vercel |
| 110 | ui.com | Score 74 out of 100, grade C | all allowed | Next.js . Amazon CloudFront |
| 111 | reddit.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 114 | chatgpt.com | Score 14 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 123 | samsung.com | Score 86 out of 100, grade B | all allowed | Adobe Experience Manager . Akamai |
| 124 | webex.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 125 | openai.com | Score 64 out of 100, grade C | all allowed | Contentful . Cloudflare |
| 132 | dropbox.com | Score 83 out of 100, grade B | all allowed | unidentified |
| 133 | android.com | Score 67 out of 100, grade C | all allowed | unidentified |
| 137 | outlook.com | Score 45 out of 100, grade D | all allowed | unidentified |
| 153 | tumblr.com | Score 54 out of 100, grade D | 4 blocked | WordPress |
| 154 | crashlytics.com | Score 60 out of 100, grade C | all allowed | Google Cloud / GFE |
| 155 | nytimes.comrefused GPTBot | Score 21 out of 100, grade F, partial assessment | 11 blocked | Fastly |
| 159 | snapchat.com | Score 46 out of 100, grade D | 5 blocked | Next.js . Google Cloud / GFE |
| 160 | samsungcloud.com | Score 52 out of 100, grade D | all allowed | Next.js . Amazon CloudFront |
| 161 | ubuntu.com | Score 83 out of 100, grade B | all allowed | unidentified |
| 162 | vk.com | Score 48 out of 100, grade D | all allowed | unidentified |
| 164 | meraki.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 167 | gravatar.com | Score 60 out of 100, grade C | 2 blocked | WordPress |
| 168 | shopify.com | Score 73 out of 100, grade C | all allowed | Shopify . Cloudflare |
| 171 | godaddy.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 173 | amazon-adsystem.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | unidentified |
| 175 | paypal.comrefused GPTBot | Score 92 out of 100, grade A, partial assessment | all allowed | Fastly |
| 179 | firefox-portal-detection.com | Score 78 out of 100, grade B, partial assessment | all allowed | Fastly |
| 181 | flickr.com | Score 42 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 182 | discord.com | Score 77 out of 100, grade B | all allowed | Webflow . Cloudflare |
| 183 | forter.com | Score 87 out of 100, grade B | 1 blocked | WordPress . Cloudflare |
| 192 | cnn.com | Score 31 out of 100, grade F | 9 blocked | Fastly |
| 196 | kaspersky.com | Score 86 out of 100, grade B | all allowed | Next.js |
| 198 | ebay.com | Score 60 out of 100, grade C, partial assessment | 5 blocked | Akamai |
| 200 | brave.com | Score 75 out of 100, grade B | all allowed | Amazon CloudFront |
| 202 | soundcloud.com | Score 31 out of 100, grade F, partial assessment | 10 blocked | Amazon CloudFront |
| 203 | theguardian.com | Score 46 out of 100, grade D | 6 blocked | unidentified |
| 206 | unity3d.com | Score 79 out of 100, grade B | all allowed | Sanity |
| 209 | medium.comrefused GPTBot | Score 46 out of 100, grade D | 4 blocked | Cloudflare |
| 210 | hcaptcha.com | Score 81 out of 100, grade B | all allowed | Cloudflare |
| 212 | stripe.com | Score 80 out of 100, grade B | all allowed | Next.js |
| 213 | roku.com | Score 77 out of 100, grade B | all allowed | Next.js . Cloudflare |
| 214 | publicnode.com | Score 59 out of 100, grade D | all allowed | Next.js . Cloudflare |
| 215 | oracle.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 219 | bbc.com | Score 31 out of 100, grade F | 10 blocked | Next.js . Fastly |
| 222 | afternic.com | Score 60 out of 100, grade C | all allowed | Contentful . Akamai |
| 226 | weather.com | Score 82 out of 100, grade B | all allowed | Next.js . Akamai |
| 227 | forbes.comrefused GPTBot | Score 57 out of 100, grade D, partial assessment | 5 blocked | Fastly |
| 229 | namecheap.comrefused GPTBot | Score 92 out of 100, grade A, partial assessment | all allowed | Cloudflare |
| 231 | sciencedirect.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 234 | xiaomi.com | Score 60 out of 100, grade C | all allowed | Akamai |
| 243 | cisco.com | Score 92 out of 100, grade A, partial assessment | all allowed | Akamai |
| 245 | applovin.com | Score 79 out of 100, grade B | all allowed | Next.js . Google Cloud / GFE |
| 249 | dnsmadeeasy.com | Score 55 out of 100, grade D | all allowed | Adobe Experience Manager . Fastly |
| 254 | centinel-analytica.com | Score 88 out of 100, grade B | all allowed | Next.js . Cloudflare |
| 257 | salesforce.com | Score 77 out of 100, grade B | all allowed | Contentful . Akamai |
| 263 | hubspot.com | Score 82 out of 100, grade B | all allowed | HubSpot CMS . Cloudflare |
| 265 | imdb.com | Score 9 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 266 | googleblog.com | Score 57 out of 100, grade D | all allowed | Google Cloud / GFE |
| 267 | alidns.com | Score 55 out of 100, grade D | all allowed | Alibaba Cloud CDN |
| 269 | arubanetworks.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 272 | icanhazip.com | Score 84 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 278 | launchdarkly.com | Score 73 out of 100, grade C | all allowed | Next.js . Netlify |
| 279 | slack.com | Score 75 out of 100, grade B | all allowed | unidentified |
| 283 | ibm.com | Score 73 out of 100, grade C, partial assessment | all allowed | Adobe Experience Manager . Akamai |
| 284 | hp.com | Score 62 out of 100, grade C | all allowed | Adobe Experience Manager . Akamai |
| 285 | reuters.com | Score 26 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 291 | booking.com | Score 92 out of 100, grade A, partial assessment | all allowed | Amazon CloudFront |
| 293 | withgoogle.comrefused GPTBot | Score 42 out of 100, grade D | all allowed | Google Cloud / GFE |
| 294 | nature.com | Score 45 out of 100, grade D, partial assessment | 7 blocked | Fastly |
| 297 | steampowered.com | Score 55 out of 100, grade D | all allowed | unidentified |
| 298 | canva.comrefused GPTBot | Score 66 out of 100, grade C, partial assessment | 4 blocked | Cloudflare |
| 299 | gamepass.com | Score 52 out of 100, grade D | all allowed | unidentified |
| 304 | ip-api.com | Score 60 out of 100, grade C | all allowed | unidentified |
| 310 | grammarly.com | Score 64 out of 100, grade C | all allowed | Contentful . Amazon CloudFront |
| 311 | springer.com | Score 92 out of 100, grade A, partial assessment | all allowed | Fastly |
| 316 | zendesk.com | Score 86 out of 100, grade B | all allowed | Next.js . Cloudflare |
| 317 | miui.com | Score 52 out of 100, grade D | all allowed | Next.js |
| 321 | wiley.comrefused GPTBot | Score 47 out of 100, grade D | all allowed | Cloudflare |
| 322 | anthropic.com | Score 64 out of 100, grade C | all allowed | Next.js . Cloudflare |
| 323 | avast.com | Score 62 out of 100, grade C | all allowed | Adobe Experience Manager . Fastly |
| 325 | vungle.com | Score 76 out of 100, grade B | all allowed | WordPress . Cloudflare |
| 327 | alibaba.com | Score 53 out of 100, grade D | 2 blocked | Alibaba Cloud CDN |
| 331 | weibo.com | Score 18 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 340 | espn.com | Score 72 out of 100, grade C, partial assessment | 3 blocked | Amazon CloudFront |
| 341 | trustpilot.com | Score 66 out of 100, grade C, partial assessment | 2 blocked | unidentified |
| 343 | autodesk.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 345 | etsy.comrefused GPTBot | Score 84 out of 100, grade B, partial assessment | all allowed | Fastly |
| 346 | paloaltonetworks.com | Score 71 out of 100, grade C | all allowed | Akamai |
| 347 | unisys.com | Score 83 out of 100, grade B | all allowed | Cloudflare |
| 350 | smartadserver.com | Score 55 out of 100, grade D | all allowed | WordPress . Cloudflare |
| 351 | naver.com | Score 6 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 352 | okta.com | Score 77 out of 100, grade B | all allowed | Adobe Experience Manager . Cloudflare |
| 353 | ea.com | Score 66 out of 100, grade C | 1 blocked | Next.js |
| 355 | tp-link.com | Score 74 out of 100, grade C | all allowed | Amazon CloudFront |
| 356 | bilibili.comrefused GPTBot | Score 57 out of 100, grade D | all allowed | unidentified |
| 357 | userapi.com | Score 48 out of 100, grade D | all allowed | unidentified |
| 358 | nvidia.com | Score 82 out of 100, grade B | all allowed | Adobe Experience Manager |
| 359 | tradingview.com | Score 74 out of 100, grade C | all allowed | Amazon CloudFront |
| 361 | dynatrace.com | Score 89 out of 100, grade B | all allowed | Nuxt . Amazon CloudFront |
| 363 | dell.com | Score 80 out of 100, grade B | all allowed | Adobe Experience Manager . Akamai |
| 366 | dotomi.com | Score 69 out of 100, grade C | all allowed | Next.js . Amazon CloudFront |
| 367 | checkpoint.comrefused GPTBot | Score 70 out of 100, grade C | all allowed | WordPress . Amazon CloudFront |
| 369 | mi.com | Score 60 out of 100, grade C | all allowed | Akamai |
| 370 | elasticbeanstalk.com | Score 67 out of 100, grade C | all allowed | Amazon CloudFront |
| 371 | wsj.com | Score 31 out of 100, grade F, partial assessment | 8 blocked | unidentified |
| 372 | weebly.com | Score 69 out of 100, grade C | all allowed | Cloudflare |
| 376 | aliyun.com | Score 55 out of 100, grade D | all allowed | Alibaba Cloud CDN |
| 378 | browser-intake-datadoghq.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | unidentified |
| 380 | netease.com | Score 52 out of 100, grade D | all allowed | unidentified |
| 385 | gmail.com | Score 64 out of 100, grade C | all allowed | Google Cloud / GFE |
| 388 | xboxlive.com | Score 55 out of 100, grade D | all allowed | unidentified |
| 392 | globo.com | Score 56 out of 100, grade D | 2 blocked | unidentified |
| 393 | go.com | Score 56 out of 100, grade D | 2 blocked | Amazon CloudFront |
| 394 | wps.com | Score 81 out of 100, grade B | all allowed | Nuxt . Amazon CloudFront |
| 397 | name.com | Score 80 out of 100, grade B | 1 blocked | Sanity . Cloudflare |
| 399 | gitlab.com | Score 73 out of 100, grade C | all allowed | Nuxt . Cloudflare |
| 401 | amazontrust.com | Score 52 out of 100, grade D | all allowed | Amazon CloudFront |
| 402 | issuu.com | Score 64 out of 100, grade C | all allowed | Next.js . Fastly |
| 404 | bloomberg.comrefused GPTBot | Score 27 out of 100, grade F, partial assessment | 10 blocked | Fastly |
| 407 | taobao.com | Score 55 out of 100, grade D | all allowed | Alibaba Cloud CDN |
| 409 | opendns.com | Score 58 out of 100, grade D | all allowed | unidentified |
| 418 | plesk.com | Score 79 out of 100, grade B | all allowed | WordPress . Cloudflare |
| 420 | yandex.com | Score 56 out of 100, grade D | all allowed | unidentified |
| 421 | visualstudio.com | Score 71 out of 100, grade C | all allowed | WordPress |
| 423 | calendly.com | Score 82 out of 100, grade B | all allowed | Astro . Cloudflare |
| 424 | firetvcaptiveportal.com | Score 78 out of 100, grade B, partial assessment | all allowed | unidentified |
| 426 | substack.com | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 427 | duckduckgo.com | Score 64 out of 100, grade C | all allowed | Next.js |
| 428 | 163.comrefused GPTBot | Score 51 out of 100, grade D | all allowed | Alibaba Cloud CDN |
| 429 | temu.com | Score 74 out of 100, grade C | all allowed | unidentified |
| 430 | aliexpress.com | Score 71 out of 100, grade C, partial assessment | 3 blocked | Alibaba Cloud CDN |
| 433 | aol.com | Score 50 out of 100, grade D | 3 blocked | Next.js . Amazon CloudFront |
| 435 | steamcommunity.com | Score 55 out of 100, grade D | all allowed | unidentified |
| 437 | indeed.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 438 | appsflyer.com | Score 95 out of 100, grade A | all allowed | WordPress . Cloudflare |
| 443 | expireddomains.comrefused GPTBot | Score 84 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 448 | dailymotion.com | Score 57 out of 100, grade D | all allowed | Amazon CloudFront |
| 455 | mcafee.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 456 | intel.com | Score 92 out of 100, grade A, partial assessment | all allowed | Akamai |
| 457 | mzstatic.com | Score 70 out of 100, grade C | all allowed | unidentified |
| 465 | duolingo.com | Score 52 out of 100, grade D | all allowed | Amazon CloudFront |
| 467 | pixabay.comrefused GPTBot | Score 67 out of 100, grade C, partial assessment | 4 blocked | Cloudflare |
| 470 | bluehost.com | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 471 | braze.com | Score 73 out of 100, grade C | all allowed | Sanity . Cloudflare |
| 472 | trendmicro.com | Score 77 out of 100, grade B | all allowed | Adobe Experience Manager |
| 477 | dropcatch.com | Score 48 out of 100, grade D | all allowed | Cloudflare |
| 478 | shein.com | Score 71 out of 100, grade C | all allowed | Cloudflare |
| 479 | goodreads.com | Score 56 out of 100, grade D | 1 blocked | Amazon CloudFront |
| 480 | teamviewer.comrefused GPTBot | Score 64 out of 100, grade C | all allowed | Next.js . Cloudflare |
| 481 | walmart.com | Score 74 out of 100, grade C | all allowed | Next.js |
| 483 | huawei.com | Score 71 out of 100, grade C | all allowed | Sitecore |
| 486 | cnbc.comrefused GPTBot | Score 24 out of 100, grade F | 11 blocked | Akamai |
| 490 | markmonitor.comrefused GPTBot | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 492 | businessinsider.com | Score 72 out of 100, grade C | 2 blocked | Fastly |
| 493 | wp.comrefused GPTBot | Score 77 out of 100, grade B | all allowed | WordPress |
| 497 | eventbrite.com | Score 64 out of 100, grade C | all allowed | Next.js . Amazon CloudFront |
| 499 | hostgator.com | Score 78 out of 100, grade B, partial assessment | all allowed | Cloudflare |
| 500 | scribd.com | Score 78 out of 100, grade B, partial assessment | all allowed | Next.js . Fastly |
| 501 | tplinkcloud.com | Score 52 out of 100, grade D | all allowed | Amazon CloudFront |
| 503 | foxnews.com | Score 89 out of 100, grade B | all allowed | Fastly |
| 504 | quora.com | Score 13 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 505 | oraclecloud.com | Score 78 out of 100, grade B, partial assessment | all allowed | Akamai |
| 506 | lijit.com | Score 49 out of 100, grade D | all allowed | Amazon CloudFront |
| 509 | statista.comrefused GPTBot | Score 43 out of 100, grade D | 7 blocked | Amazon CloudFront |
| 512 | playstation.com | Score 73 out of 100, grade C | all allowed | Adobe Experience Manager . Akamai |
| 514 | addtoany.com | Score 55 out of 100, grade D | all allowed | Cloudflare |
Showing the 200 most-visited.
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Agent readiness on .com." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.