Change feed
What moved recently
Robots.txt edits are never announced. This is every movement the crawler has detected, newest first. A domain's first measurement is a baseline and is not listed, and comparisons are never made across a probe change or a change of crawl location.
| Date | Type | Domain | What changed |
|---|---|---|---|
| Score | zscaler.com | Score fell 6 points to 77. | |
| Agent files | zscaler.com | zscaler.com: llms.txt removed. | |
| Score | zoho.eu | Score fell 3 points to 57. | |
| Score | zoho.com | Score fell 3 points to 57. | |
| Score | zillow.com | Score fell 28 points to 50. | |
| Availability | zaloapp.com | Went unreachable: ETIMEDOUT. | |
| Availability | zalo.me | Went unreachable: ETIMEDOUT. | |
| Agent files | zalando.de | zalando.de: llms.txt added. | |
| Crawler policy | zalando.de | zalando.de now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended. | |
| Availability | zalando.de | Came back online. | |
| Availability | ys7.com | Went unreachable: homepage returned HTTP 404. | |
| Score | youradchoices.com | Score rose 6 points to 84. | |
| Availability | youku.com | Went unreachable: ETIMEDOUT. | |
| Score | yieldmo.com | Score rose 14 points to 92. | |
| Agent files | yieldmo.com | yieldmo.com: llms.txt added. | |
| Availability | ydparty06.tv | Came back online. | |
| Availability | yastatic.net | Went unreachable: ETIMEDOUT. | |
| Availability | yandex.ru | Went unreachable: ETIMEDOUT. | |
| Availability | yandex.net | Went unreachable: ETIMEDOUT. | |
| Availability | yandex.com.tr | Went unreachable: ETIMEDOUT. | |
| Availability | yale.edu | Went unreachable: ERR_TLS_CERT_ALTNAME_INVALID. | |
| Score | xxx18hot.com | Score rose 3 points to 57. | |
| Score | xnxxvideos.me | Score rose 36 points to 84. | |
| Availability | xnhau.you | Went unreachable: timeout. | |
| Score | xnhau.xxx | Score fell 48 points to 30. | |
| Crawler policy | xnhau.xxx | xnhau.xxx now blocks GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | xnhau.show | Score fell 46 points to 32. | |
| Crawler policy | xnhau.show | xnhau.show now blocks GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Availability | xnhau.ink | Went unreachable: timeout. | |
| Score | xiaomi.net | Score rose 4 points to 64. | |
| Score | xiaomi.com | Score rose 4 points to 64. | |
| Score | xi-rutor.top | Score fell 40 points to 52. | |
| Score | xe.com | Score fell 3 points to 86. | |
| Score | wunderground.com | Score fell 12 points to 50. | |
| Score | wetteronline.de | Score rose 20 points to 78. | |
| Score | wetransfer.com | Score rose 7 points to 61. | |
| Score | westernunion.com | Score fell 3 points to 77. | |
| Score | vk.ru | Score fell 36 points to 48. | |
| Score | vivo.com | Score rose 7 points to 56. | |
| Score | visa.com | Score rose 7 points to 64. | |
| Score | virtualmin.com | Score fell 7 points to 57. | |
| Score | uxfeedback.ru | Score fell 4 points to 57. | |
| Score | usps.com | Score fell 3 points to 68. | |
| Availability | usp.br | Went unreachable: timeout. | |
| Score | usercentrics.eu | Score fell 7 points to 84. | |
| Agent files | upstart.com | upstart.com: llms.txt removed. | |
| Score | unesco.org | Score rose 5 points to 68. | |
| Crawler policy | unesco.org | unesco.org now allows Google-Extended. | |
| Score | un.org | Score fell 3 points to 45. | |
| Score | umass.edu | Score rose 14 points to 78. | |
| Availability | uidai.gov.in | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | udn.com | Score rose 7 points to 81. | |
| Agent files | ucsf.edu | ucsf.edu: llms.txt removed. | |
| Score | uchicago.edu | Score fell 3 points to 64. | |
| Availability | uc.cn | Went unreachable: ETIMEDOUT. | |
| Score | twkan.com | Score fell 46 points to 32. | |
| Crawler policy | twkan.com | twkan.com now blocks GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Availability | tvmon1.com | Went unreachable: ETIMEDOUT. | |
| Score | turbo.az | Score fell 23 points to 55. | |
| Score | tumblr.com | Score fell 4 points to 41. | |
| Score | tum.de | Score rose 9 points to 89. | |
| Agent files | tum.de | tum.de: llms.txt added. | |
| Score | trulia.com | Score fell 23 points to 69. | |
| Availability | truecaller.com | Went unreachable: ETIMEDOUT. | |
| Score | transip.nl | Score fell 7 points to 67. | |
| Availability | tmall.com | Went unreachable: ETIMEDOUT. | |
| Score | tiktokio.com | Score rose 12 points to 92. | |
| Score | thisvid.com | Score rose 3 points to 53. | |
| Score | thestar.com | Score rose 20 points to 34. | |
| Score | theqoo.net | Score rose 21 points to 78. | |
| Agent files | themegrill.com | themegrill.com: llms.txt added. | |
| Score | tfl.gov.uk | Score fell 12 points to 66. | |
| Score | tencent.com | Score fell 3 points to 66. | |
| Score | temu.com | Score rose 4 points to 78. | |
| Availability | telstra.com | Went unreachable: ETIMEDOUT. | |
| Score | teamviewer.com | Score rose 3 points to 67. | |
| Score | teacherspayteachers.com | Score rose 28 points to 78. | |
| Crawler policy | teacherspayteachers.com | teacherspayteachers.com now allows GPTBot, Applebot-Extended, meta-externalagent. | |
| Availability | tbank.ru | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | tappx.com | Score rose 60 points to 66. | |
| Crawler policy | tappx.com | tappx.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Availability | taobao.com | Went unreachable: ETIMEDOUT. | |
| Score | t-mobile.com | Score rose 24 points to 78. | |
| Score | syosetu.org | Score rose 24 points to 78. | |
| Crawler policy | syosetu.org | syosetu.org now allows GPTBot, Google-Extended. | |
| Availability | swiftkey.com | Went unreachable: homepage returned HTTP 404. | |
| Score | ssstik.io | Score rose 16 points to 92. | |
| Score | sports.ru | Score rose 25 points to 78. | |
| Score | speedtest.net | Score rose 27 points to 84. | |
| Score | skyscanner.net | Score fell 41 points to 45. | |
| Score | siteground.biz | Score rose 14 points to 92. | |
| Agent files | siteground.biz | siteground.biz: llms.txt added. | |
| Availability | sinoptik.ua | Came back online. | |
| Score | siemens.com | Score fell 7 points to 69. | |
| Score | sicredi.com.br | Score rose 3 points to 59. | |
| Score | shopee.com | Score fell 14 points to 41. | |
| Score | sex-studentki.team | Score fell 6 points to 78. | |
| Score | sex-studentki.studio | Score fell 6 points to 78. | |
| Score | sex-studentki.cafe | Score fell 6 points to 78. | |
| Availability | servidoresdns.net | Came back online. | |
| Score | senderscore.com | Score fell 19 points to 73. | |
| Score | senate.gov | Score rose 26 points to 78. | |
| Availability | sedo.com | Went unreachable: homepage returned HTTP 523. | |
| Score | searchenginejournal.com | Score rose 9 points to 83. | |
| Agent files | searchenginejournal.com | searchenginejournal.com: llms.txt added. | |
| Score | seagate.com | Score rose 23 points to 92. | |
| Availability | screener.in | Went unreachable: ETIMEDOUT. | |
| Score | savefrom.net | Score rose 11 points to 77. | |
| Availability | sattamatkadpboss.mobi | Came back online. | |
| Score | rusprofile.ru | Score fell 7 points to 50. | |
| Score | roku.com | Score fell 7 points to 70. | |
| Score | ring.com | Score fell 12 points to 68. | |
| Score | richaudience.com | Score fell 3 points to 55. | |
| Score | reolink.com | Score fell 3 points to 64. | |
| Agent files | razorpay.com | razorpay.com: llms.txt added, agents.md added. | |
| Availability | razorpay.com | Came back online. | |
| Score | rambler.ru | Score fell 3 points to 64. | |
| Availability | rajasthan.gov.in | Went unreachable: ETIMEDOUT. | |
| Availability | qunar.com | Went unreachable: ETIMEDOUT. | |
| Availability | quark.cn | Went unreachable: ETIMEDOUT. | |
| Availability | qcloudcdn.com | Went unreachable: ETIMEDOUT. | |
| Score | pwc.com | Score rose 11 points to 78. | |
| Score | publi24.ro | Score rose 12 points to 78. | |
| Agent files | publi24.ro | publi24.ro: llms.txt removed. | |
| Score | ptgaming.ph | Score rose 24 points to 78. | |
| Availability | ps.kz | Went unreachable: ETIMEDOUT. | |
| Score | prokerala.com | Score fell 10 points to 62. | |
| Score | primevideo.com | Score fell 3 points to 37. | |
| Score | priceline.com | Score fell 21 points to 63. | |
| Score | powerapps.com | Score fell 3 points to 52. | |
| Availability | pornpics.com | Went unreachable: ECONNRESET. | |
| Score | popularmechanics.com | Score fell 7 points to 60. | |
| Availability | pmkisan.gov.in | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | plex.tv | Score fell 3 points to 60. | |
| Score | playstation.com | Score rose 3 points to 73. | |
| Score | pin.it | Score rose 7 points to 48. | |
| Score | photobucket.com | Score fell 3 points to 45. | |
| Score | pewresearch.org | Score fell 9 points to 79. | |
| Score | perplexity.ai | Score fell 14 points to 78. | |
| Score | permutive.com | Score fell 7 points to 73. | |
| Score | perl.org | Score fell 3 points to 55. | |
| Score | people.com.cn | Score fell 7 points to 52. | |
| Score | people.cn | Score fell 7 points to 52. | |
| Score | penguinrandomhouse.com | Score rose 54 points to 60. | |
| Crawler policy | penguinrandomhouse.com | penguinrandomhouse.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | pendo.io | Score fell 7 points to 79. | |
| Score | peatix.com | Score rose 9 points to 61. | |
| Score | peacocktv.com | Score fell 9 points to 52. | |
| Score | pcworld.com | Score rose 21 points to 78. | |
| Crawler policy | pcworld.com | pcworld.com now allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | pcre.org | Score rose 6 points to 84. | |
| Score | pcmag.com | Score rose 43 points to 78. | |
| Crawler policy | pcmag.com | pcmag.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Applebot-Extended, meta-externalagent. | |
| Score | paypal.com | Score fell 6 points to 80. | |
| Availability | pavietnam.vn | Went unreachable: ETIMEDOUT. | |
| Score | oxfordjournals.org | Score rose 14 points to 92. | |
| Crawler policy | ovh.com | ovh.com now allows GPTBot, ChatGPT-User. | |
| Availability | ovh.com | Went unreachable: homepage returned HTTP 404. | |
| Availability | osym.gov.tr | Came back online. | |
| Score | oppo.com | Score fell 10 points to 70. | |
| Availability | opendns.com | Came back online. | |
| Score | nustargame.com | Score rose 32 points to 92. | |
| Availability | ntv.co.jp | Went unreachable: ETIMEDOUT. | |
| Score | norton.com | Score fell 11 points to 71. | |
| Agent files | norton.com | norton.com: llms.txt removed. | |
| Score | nintendo.com | Score rose 10 points to 58. | |
| Score | nike.com | Score rose 15 points to 92. | |
| Availability | nexx360.io | Went unreachable: timeout. | |
| Availability | newtoki1.org | Went unreachable: ETIMEDOUT. | |
| Availability | mogujie.com | Went unreachable: ETIMEDOUT. | |
| Score | mobilityware.com | Score rose 14 points to 92. | |
| Score | miwifi.com | Score fell 29 points to 49. | |
| Availability | mingky08.tv | Came back online. | |
| Availability | milfporn.tv | Came back online. | |
| Score | microsoft.com | Score fell 3 points to 62. | |
| Score | mi.com | Score rose 4 points to 64. | |
| Crawler policy | meteociel.fr | meteociel.fr now blocks ClaudeBot. | |
| Availability | meteociel.fr | Came back online. | |
| Score | menshealth.com | Score fell 7 points to 60. | |
| Score | meethue.com | Score fell 3 points to 74. | |
| Score | mapquest.com | Score rose 19 points to 64. | |
| Crawler policy | mapquest.com | mapquest.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | mangabuff.ru | Score fell 13 points to 65. | |
| Score | lyft.com | Score fell 7 points to 64. | |
| Score | lowes.com | Score rose 16 points to 78. | |
| Score | loobygamesonline.com | Score fell 3 points to 54. | |
| Score | logitech.com | Score fell 10 points to 67. | |
| Score | locaweb.com.br | Score rose 3 points to 58. | |
| Score | loc.gov | Score rose 21 points to 78. | |
| Agent files | lge.com | lge.com: llms.txt removed. | |
| Availability | lge.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | lg.com | Score rose 9 points to 83. | |
| Score | leroymerlin.fr | Score fell 14 points to 78. | |
| Availability | lenovo.com.cn | Went unreachable: ERR_TLS_CERT_ALTNAME_INVALID. | |
| Score | lenovo.com | Score rose 6 points to 60. | |
| Score | legacy.com | Score rose 26 points to 92. | |
| Availability | lausd.net | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | lastpass.com | Score fell 7 points to 67. | |
| Score | ladbrokes.com | Score rose 31 points to 78. | |
| Score | klaviyo.com | Score fell 3 points to 86. | |
| Availability | kinopoisk.ru | Went unreachable: ETIMEDOUT. | |
| Availability | kinnser.net | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | kia.com | Score fell 13 points to 59. | |
| Availability | kemenag.go.id | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Availability | kaiserpermanente.org | Came back online. | |
| Score | justia.com | Score fell 14 points to 78. | |
| Score | joom.com | Score rose 9 points to 54. | |
| Score | join.chat | Score fell 23 points to 55. | |
| Crawler policy | join.chat | join.chat now blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Agent files | jio.com | jio.com: llms.txt removed. | |
| Crawler policy | jio.com | jio.com now allows meta-externalagent. | |
| Availability | jio.com | Went unreachable: ETIMEDOUT. | |
| Availability | jftechws.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | jennymovies.com | Score rose 4 points to 78. | |
| Score | jd.com | Score fell 29 points to 55. | |
| Score | japantimes.co.jp | Score rose 15 points to 78. | |
| Availability | ixbt.com | Came back online. | |
| Score | it.com | Score rose 21 points to 78. | |
| Crawler policy | iqiyi.com | iqiyi.com now allows GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot. | |
| Availability | iqiyi.com | Went unreachable: ETIMEDOUT. | |
| Score | indiamart.com | Score rose 27 points to 77. | |
| Score | imageshack.us | Score fell 6 points to 13. | |
| Score | ign.com | Score rose 6 points to 33. | |
| Score | ifood.com.br | Score rose 14 points to 92. | |
| Score | hyundai.com | Score rose 9 points to 77. | |
| Score | hulu.com | Score fell 23 points to 54. | |
| Crawler policy | hulu.com | hulu.com now allows Perplexity-User. | |
| Availability | huawei.com | Went unreachable: ETIMEDOUT. | |
| Score | hrw.org | Score fell 21 points to 63. | |
| Score | hp.com | Score rose 10 points to 62. | |
| Availability | hosteurope.de | Came back online. | |
| Score | homedepot.com | Score fell 14 points to 78. | |
| Score | home.pl | Score fell 3 points to 58. | |
| Score | history.com | Score fell 20 points to 55. | |
| Availability | hinet.net | Went unreachable: ETIMEDOUT. | |
| Score | harpersbazaar.com | Score fell 7 points to 57. | |
| Score | hardened-php.net | Score rose 14 points to 92. | |
| Availability | guardingvision.com | Went unreachable: ENOTFOUND. | |
| Availability | gst.gov.in | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | google.tm | Score fell 3 points to 49. | |
| Score | google.pl | Score fell 3 points to 49. | |
| Score | google.nl | Score fell 3 points to 49. | |
| Score | google.mw | Score fell 3 points to 49. | |
| Score | google.mv | Score rose 3 points to 52. | |
| Score | google.it | Score rose 3 points to 52. | |
| Score | google.im | Score fell 3 points to 49. | |
| Score | google.gm | Score fell 3 points to 49. | |
| Score | google.fi | Score rose 3 points to 52. | |
| Score | google.es | Score rose 3 points to 52. | |
| Score | google.dk | Score rose 3 points to 52. | |
| Score | google.cz | Score rose 3 points to 52. | |
| Score | google.com.vn | Score fell 3 points to 49. | |
| Score | google.com.tr | Score rose 3 points to 52. | |
| Score | google.com.py | Score fell 3 points to 49. | |
| Score | google.com.pk | Score rose 3 points to 52. | |
| Score | google.com.ph | Score fell 3 points to 49. | |
| Score | google.com.pg | Score rose 3 points to 52. | |
| Score | google.com.ng | Score rose 3 points to 52. | |
| Score | google.com.na | Score rose 3 points to 52. | |
| Score | google.com.mx | Score rose 3 points to 52. | |
| Score | google.com.hk | Score rose 3 points to 52. | |
| Score | google.com.eg | Score fell 3 points to 49. | |
| Score | google.com.do | Score rose 3 points to 52. | |
| Score | google.com.br | Score rose 3 points to 52. | |
| Score | google.com.bd | Score rose 3 points to 52. | |
| Score | google.com.au | Score rose 3 points to 52. | |
| Score | google.com.ar | Score rose 3 points to 52. | |
| Score | google.co.za | Score fell 3 points to 49. | |
| Score | google.co.ve | Score rose 3 points to 52. | |
| Score | google.co.th | Score rose 3 points to 52. | |
| Score | google.co.nz | Score rose 3 points to 52. | |
| Score | google.co.kr | Score rose 3 points to 52. | |
| Score | google.co.id | Score rose 3 points to 52. | |
| Score | google.cl | Score rose 3 points to 52. | |
| Score | goodhousekeeping.com | Score fell 7 points to 69. | |
| Score | gnu.org | Score fell 7 points to 54. | |
| Score | gitee.com | Score fell 7 points to 63. | |
| Score | gettyimages.com | Score fell 7 points to 64. | |
| Score | gazeta.pl | Score fell 4 points to 53. | |
| Score | garmin.com | Score fell 3 points to 55. | |
| Availability | garena.com | Went unreachable: timeout. | |
| Score | garant.ru | Score rose 6 points to 92. | |
| Score | gap.com | Score fell 9 points to 62. | |
| Score | footmercato.net | Score rose 3 points to 80. | |
| Score | foodnetwork.com | Score fell 14 points to 78. | |
| Score | fitbit.com | Score rose 8 points to 65. | |
| Score | findagrave.com | Score rose 31 points to 84. | |
| Score | filmywap.technology | Score rose 7 points to 59. | |
| Score | fidelity.com | Score rose 15 points to 92. | |
| Score | feishu.cn | Score fell 4 points to 48. | |
| Score | faqs.org | Score rose 26 points to 78. | |
| Score | fapello.com | Score fell 6 points to 78. | |
| Score | fang.com | Score rose 3 points to 55. | |
| Score | fanduel.com | Score fell 5 points to 78. | |
| Agent files | fanduel.com | fanduel.com: llms.txt removed. | |
| Availability | ezvizlife.com | Went unreachable: homepage returned HTTP 503. | |
| Score | esquire.com | Score fell 7 points to 60. | |
| Score | eset.com | Score fell 10 points to 64. | |
| Availability | esa.int | Came back online. | |
| Score | elpais.com | Score rose 5 points to 54. |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Changes in AI crawler policy." https://crawlindex.org (measured 2026-09-22). Licensed CC BY 4.0.