Change feed
What moved recently
Robots.txt edits are never announced. This is every movement the crawler has detected, newest first. A domain's first measurement is a baseline and is not listed, and comparisons are never made across a probe change or a change of crawl location.
| Date | Type | Domain | What changed |
|---|---|---|---|
| Score | zillow.com | Score fell 28 points to 50. | |
| Agent files | zalando.de | zalando.de: llms.txt added. | |
| Availability | zalando.de | Came back online. | |
| Availability | ys7.com | Went unreachable: homepage returned HTTP 404. | |
| Score | youku.com | Score fell 26 points to 52. | |
| Score | xhzone.live | Score fell 12 points to 48. | |
| Score | xhtv.live | Score fell 12 points to 48. | |
| Score | xhstream.com | Score fell 12 points to 48. | |
| Score | xhspot.com | Score fell 12 points to 48. | |
| Score | xhsocial.com | Score fell 12 points to 48. | |
| Score | xhopen.com | Score fell 12 points to 48. | |
| Score | xhname.com | Score fell 12 points to 48. | |
| Score | xhlink.blog | Score fell 12 points to 48. | |
| Score | xhfuck.com | Score fell 12 points to 48. | |
| Score | xhfanclub.com | Score fell 12 points to 48. | |
| Score | xhco.live | Score fell 12 points to 48. | |
| Score | xhamster46.desi | Score fell 12 points to 48. | |
| Score | xhamster2.com | Score fell 12 points to 48. | |
| Score | xhamster19.com | Score fell 12 points to 48. | |
| Score | xhamster1.desi | Score fell 12 points to 48. | |
| Score | xhamster.desi | Score fell 12 points to 48. | |
| Score | xhamster.com | Score fell 12 points to 48. | |
| Score | xhaccess.com | Score fell 12 points to 48. | |
| Score | vivo.com | Score fell 7 points to 49. | |
| Availability | visa.cn | Went unreachable: ETIMEDOUT. | |
| Score | vigo.tech | Score rose 7 points to 64. | |
| Score | unruly.co | Score fell 7 points to 66. | |
| Availability | unhcr.org | Came back online. | |
| Score | umeng.com | Score rose 6 points to 80. | |
| Score | tumblr.com | Score rose 4 points to 54. | |
| Score | trulia.com | Score fell 23 points to 69. | |
| Availability | tokyomotion.net | Came back online. | |
| Score | time.com | Score fell 6 points to 67. | |
| Agent files | time.com | time.com: llms.txt removed. | |
| Score | tilda.ws | Score fell 23 points to 55. | |
| Availability | thisvid.com | Went unreachable: homepage returned HTTP 502. | |
| Score | thestar.com | Score rose 3 points to 34. | |
| Score | sxyprn.com | Score rose 29 points to 84. | |
| Score | stbid.ru | Score rose 72 points to 78. | |
| Crawler policy | stbid.ru | stbid.ru now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | splunkcloud.com | Score fell 13 points to 65. | |
| Score | skyscanner.net | Score fell 41 points to 45. | |
| Score | sexwabs.com | Score fell 7 points to 56. | |
| Score | serpro.gov.br | Score fell 3 points to 58. | |
| Score | senderscore.com | Score rose 19 points to 92. | |
| Availability | sedo.com | Came back online. | |
| Score | samsclub.com | Score fell 25 points to 67. | |
| Score | rockstargames.com | Score rose 21 points to 59. | |
| Crawler policy | primevideo.com | primevideo.com now allows ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, meta-externalagent. | |
| Availability | primevideo.com | Went unreachable: homepage returned HTTP 503. | |
| Score | priceline.com | Score fell 21 points to 63. | |
| Availability | pornpics.com | Went unreachable: ECONNRESET. | |
| Score | plos.org | Score rose 14 points to 92. | |
| Score | ohmy.bid | Score rose 7 points to 59. | |
| Availability | myoffice.ru | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | muni.cz | Score fell 7 points to 59. | |
| Availability | movieswood.cloud | Went unreachable: homepage returned HTTP 523. | |
| Score | mintegral.net | Score fell 7 points to 55. | |
| Score | mintegral.com | Score fell 7 points to 55. | |
| Score | mimecast.com | Score fell 7 points to 67. | |
| Score | milfporn.tv | Score rose 4 points to 78. | |
| Availability | mileroticos.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Availability | mikrotik.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Crawler policy | kuaishou.com | kuaishou.com now blocks ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Availability | kuaishou.com | Came back online. | |
| Score | itch.io | Score fell 27 points to 65. | |
| Score | ipcc.ch | Score rose 6 points to 84. | |
| Score | intuit.com | Score rose 6 points to 92. | |
| Score | indeed.com | Score fell 6 points to 78. | |
| Score | ign.com | Score rose 3 points to 29. | |
| Score | hrw.org | Score fell 21 points to 63. | |
| Availability | hinet.net | Came back online. | |
| Availability | hawaii.edu | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | gotinder.com | Score rose 10 points to 58. | |
| Availability | googlesource.com | Went unreachable: homepage returned HTTP 503. | |
| Availability | goneo.de | Came back online. | |
| Score | gmt.io | Score rose 9 points to 86. | |
| Agent files | gmt.io | gmt.io: llms.txt added. | |
| Score | freebsd.org | Score fell 3 points to 57. | |
| Score | findagrave.com | Score rose 5 points to 19. | |
| Score | federalregister.gov | Score rose 7 points to 62. | |
| Score | faphouse2.com | Score rose 20 points to 78. | |
| Score | faphouse.com | Score rose 20 points to 78. | |
| Score | familysearch.org | Score fell 3 points to 58. | |
| Score | express.co.uk | Score fell 17 points to 61. | |
| Crawler policy | express.co.uk | express.co.uk now blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot-Extended. | |
| Score | erome.com | Score rose 3 points to 55. | |
| Score | eporner.com | Score fell 8 points to 59. | |
| Score | eenadu.net | Score fell 28 points to 42. | |
| Crawler policy | eenadu.net | eenadu.net now blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | ecfr.gov | Score rose 10 points to 62. | |
| Score | dominos.com | Score rose 3 points to 64. | |
| Score | cookiebot.com | Score fell 7 points to 81. | |
| Availability | consultant.ru | Came back online. | |
| Score | colostate.edu | Score rose 7 points to 77. | |
| Score | checkpoint.com | Score fell 7 points to 70. | |
| Score | chaturbate.com | Score fell 14 points to 78. | |
| Score | capitaloneshopping.com | Score fell 3 points to 72. | |
| Score | bu.edu | Score rose 3 points to 69. | |
| Availability | brivo.com | Came back online. | |
| Score | britishairways.com | Score rose 7 points to 70. | |
| Score | bmw-motorrad.com.cn | Score fell 7 points to 70. | |
| Score | biorxiv.org | Score rose 40 points to 78. | |
| Crawler policy | biorxiv.org | biorxiv.org now allows GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot. | |
| Score | av380.net | Score fell 36 points to 48. | |
| Agent files | arlo.com | arlo.com: llms.txt removed. | |
| Score | appfolio.com | Score fell 10 points to 71. | |
| Score | amocrm.ru | Score rose 9 points to 84. | |
| Agent files | amocrm.ru | amocrm.ru: llms.txt added. | |
| Score | alamy.com | Score fell 10 points to 62. | |
| Score | afrinic.net | Score rose 7 points to 77. | |
| Availability | afraid.org | Came back online. | |
| Availability | adotmob.com | Came back online. | |
| Score | a.co | Score rose 33 points to 78. | |
| Score | 1688.com | Score fell 36 points to 48. | |
| Score | zomato.com | Score fell 26 points to 52. | |
| Score | zhihu.com | Score fell 7 points to 42. | |
| Availability | zalando.de | Went unreachable: timeout. | |
| Agent files | zalando.de | zalando.de: llms.txt removed. | |
| Score | yumpu.com | Score rose 9 points to 54. | |
| Score | yr.no | Score fell 7 points to 31. | |
| Availability | yieldmo.com | Went unreachable: UND_ERR_SOCKET. | |
| Score | yandex.net | Score rose 3 points to 52. | |
| Score | yandex.by | Score fell 3 points to 56. | |
| Score | xiaomi.net | Score rose 7 points to 60. | |
| Score | xiaomi.com | Score rose 7 points to 60. | |
| Score | xerox.com | Score fell 7 points to 56. | |
| Score | xero.com | Score rose 7 points to 73. | |
| Score | welt.de | Score rose 7 points to 50. | |
| Score | weforum.org | Score rose 7 points to 78. | |
| Crawler policy | wayground.com | wayground.com now allows ClaudeBot. | |
| Score | washingtontimes.com | Score fell 11 points to 78. | |
| Score | wallapop.com | Score rose 24 points to 78. | |
| Score | walgreens.com | Score rose 7 points to 64. | |
| Score | voanews.com | Score rose 7 points to 67. | |
| Score | vnexpress.net | Score fell 4 points to 70. | |
| Availability | vkvideo.ru | Came back online. | |
| Score | vivox.com | Score rose 7 points to 67. | |
| Score | vivo.com | Score rose 7 points to 56. | |
| Availability | visa.cn | Came back online. | |
| Availability | virtualbox.org | Came back online. | |
| Score | vigo.tech | Score fell 7 points to 57. | |
| Score | va.gov | Score rose 13 points to 76. | |
| Score | usaa.com | Score rose 4 points to 48. | |
| Score | unity3d.com | Score rose 7 points to 79. | |
| Score | unity.com | Score rose 7 points to 79. | |
| Availability | unhcr.org | Went unreachable: ECONNRESET. | |
| Score | unfccc.int | Score fell 27 points to 57. | |
| Score | unep.org | Score rose 4 points to 65. | |
| Score | un.org | Score fell 3 points to 45. | |
| Score | umeng.com | Score fell 6 points to 74. | |
| Crawler policy | umd.edu | umd.edu now blocks GPTBot, PerplexityBot. | |
| Score | umd.edu | Score fell 14 points to 64. | |
| Score | ukdevilz.com | Score rose 6 points to 84. | |
| Score | tumblr.com | Score fell 4 points to 50. | |
| Score | trusteer.com | Score rose 4 points to 58. | |
| Score | trendmicro.com | Score rose 7 points to 77. | |
| Score | treasury.gov | Score rose 7 points to 57. | |
| Agent files | traveloka.com | traveloka.com: llms.txt removed. | |
| Score | traveloka.com | Score rose 7 points to 78. | |
| Score | torob.com | Score fell 3 points to 73. | |
| Availability | tokyomotion.net | Went unreachable: timeout. | |
| Score | today.com | Score rose 7 points to 32. | |
| Score | tizen.org | Score rose 14 points to 78. | |
| Agent files | time.com | time.com: llms.txt added. | |
| Score | time.com | Score rose 6 points to 73. | |
| Score | tilda.ws | Score rose 23 points to 78. | |
| Agent files | theweathernetwork.com | theweathernetwork.com: llms.txt removed. | |
| Score | theweathernetwork.com | Score rose 4 points to 78. | |
| Score | thestar.com | Score rose 17 points to 31. | |
| Availability | themegrill.com | Went unreachable: homepage returned HTTP 520. | |
| Score | terra.com.br | Score rose 7 points to 77. | |
| Score | telekom.de | Score rose 11 points to 78. | |
| Score | telegraph.co.uk | Score rose 5 points to 37. | |
| Agent files | teepublic.com | teepublic.com: llms.txt added. | |
| Score | syosetu.com | Score fell 26 points to 58. | |
| Score | superhosting.bg | Score fell 23 points to 55. | |
| Score | sun.com | Score rose 7 points to 60. | |
| Agent files | substack.com | substack.com: llms.txt removed. | |
| Score | substack.com | Score rose 3 points to 78. | |
| Score | subito.it | Score rose 7 points to 61. | |
| Score | studentaid.gov | Score rose 4 points to 45. | |
| Availability | stanford.edu | Came back online. | |
| Score | sportskeeda.com | Score rose 11 points to 78. | |
| Score | sotwe.com | Score rose 12 points to 90. | |
| Availability | sonobi.com | Went unreachable: homepage returned HTTP 502. | |
| Score | smithsonianmag.com | Score fell 12 points to 66. | |
| Availability | smartsheet.com | Came back online. | |
| Agent files | smartsheet.com | smartsheet.com: llms.txt added. | |
| Availability | sitescout.com | Came back online. | |
| Agent files | siteground.biz | siteground.biz: llms.txt added. | |
| Score | siteground.biz | Score rose 14 points to 92. | |
| Availability | sinoptik.ua | Came back online. | |
| Score | signifyd.com | Score rose 7 points to 89. | |
| Score | sicredi.com.br | Score rose 7 points to 59. | |
| Score | si.edu | Score fell 6 points to 78. | |
| Crawler policy | sex-studentki.vodka | sex-studentki.vodka now blocks GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Google-Extended, meta-externalagent. | |
| Score | sex-studentki.vodka | Score fell 29 points to 49. | |
| Crawler policy | sex-studentki.team | sex-studentki.team now blocks GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Google-Extended, meta-externalagent. | |
| Score | sex-studentki.team | Score fell 29 points to 49. | |
| Score | sephora.com | Score rose 7 points to 68. | |
| Crawler policy | seattletimes.com | seattletimes.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | seattletimes.com | Score rose 44 points to 78. | |
| Score | schwab.com | Score rose 7 points to 64. | |
| Score | schoology.com | Score fell 12 points to 66. | |
| Availability | sbis.ru | Came back online. | |
| Score | sbb.ch | Score rose 21 points to 78. | |
| Score | satta-king-fast.com | Score fell 30 points to 48. | |
| Score | salesforce.com | Score rose 7 points to 77. | |
| Availability | saby.ru | Came back online. | |
| Availability | ruliweb.com | Came back online. | |
| Score | rule34.xxx | Score rose 6 points to 84. | |
| Availability | rosminzdrav.ru | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | rockstargames.com | Score fell 21 points to 38. | |
| Availability | revolve.com | Went unreachable: timeout. | |
| Score | rentalcars.com | Score rose 11 points to 78. | |
| Score | redwap.sex | Score rose 33 points to 84. | |
| Score | redcross.org | Score rose 7 points to 54. | |
| Score | recruit.co.jp | Score rose 8 points to 78. | |
| Availability | rajasthan.gov.in | Came back online. | |
| Agent files | rajasthan.gov.in | rajasthan.gov.in: llms.txt added. | |
| Score | radiofrance.fr | Score fell 7 points to 39. | |
| Availability | quago.io | Went unreachable: UND_ERR_SOCKET. | |
| Score | pussyspace.com | Score fell 17 points to 36. | |
| Availability | property24.com | Came back online. | |
| Score | prnewswire.com | Score fell 25 points to 67. | |
| Availability | privacymanager.io | Went unreachable: ENOTFOUND. | |
| Agent files | privacymanager.io | privacymanager.io: llms.txt removed. | |
| Score | premierleague.com | Score fell 14 points to 64. | |
| Score | pravda.com.ua | Score fell 21 points to 71. | |
| Score | poshmark.com | Score fell 21 points to 63. | |
| Score | plex.tv | Score fell 3 points to 57. | |
| Availability | pku.edu.cn | Went unreachable: ERR_TLS_CERT_ALTNAME_INVALID. | |
| Score | pin.it | Score rose 5 points to 53. | |
| Score | pimpbunny.com | Score fell 12 points to 29. | |
| Score | pikabu.ru | Score fell 25 points to 67. | |
| Score | perplexity.ai | Score fell 47 points to 45. | |
| Score | pearson.com | Score rose 7 points to 61. | |
| Availability | pcre.org | Went unreachable: UND_ERR_SOCKET. | |
| Score | pchome.com.tw | Score fell 10 points to 68. | |
| Score | patreon.com | Score rose 37 points to 90. | |
| Score | pardot.com | Score rose 7 points to 60. | |
| Availability | over-blog.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | osu.edu | Score rose 14 points to 78. | |
| Score | openai.com | Score rose 7 points to 64. | |
| Score | ondemand.com | Score rose 7 points to 60. | |
| Score | ohmy.bid | Score fell 7 points to 52. | |
| Score | ogury.io | Score rose 7 points to 80. | |
| Score | nyu.edu | Score rose 24 points to 78. | |
| Score | nyc.gov | Score rose 7 points to 63. | |
| Score | nrk.no | Score fell 7 points to 28. | |
| Score | npr.org | Score rose 3 points to 60. | |
| Availability | novell.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Score | nokia.com | Score rose 7 points to 67. | |
| Score | njavtv.com | Score fell 23 points to 55. | |
| Score | nike.com | Score rose 7 points to 77. | |
| Score | next.co.uk | Score rose 7 points to 69. | |
| Score | nespresso.com | Score rose 3 points to 64. | |
| Score | nbcnews.com | Score rose 7 points to 30. | |
| Score | nbc.com | Score fell 5 points to 61. | |
| Score | naukri.com | Score rose 7 points to 45. | |
| Score | myportfolio.com | Score rose 30 points to 78. | |
| Score | msu.edu | Score fell 19 points to 73. | |
| Score | msdn.com | Score rose 7 points to 64. | |
| Score | moodle.com | Score rose 5 points to 78. | |
| Availability | mobilityware.com | Went unreachable: UND_ERR_SOCKET. | |
| Score | miwifi.com | Score rose 29 points to 78. | |
| Score | missav123.com | Score fell 23 points to 55. | |
| Score | missav.ws | Score fell 23 points to 55. | |
| Score | missav.live | Score fell 23 points to 55. | |
| Score | missav.ai | Score fell 23 points to 55. | |
| Score | milfporn.tv | Score fell 4 points to 74. | |
| Score | microsoft.com | Score rose 4 points to 65. | |
| Score | michaels.com | Score rose 14 points to 61. | |
| Score | mi.com | Score rose 7 points to 60. | |
| Score | medicare.gov | Score rose 7 points to 63. | |
| Agent files | mediafire.com | mediafire.com: llms.txt added. | |
| Score | medal.tv | Score rose 7 points to 71. | |
| Score | max.com | Score fell 7 points to 60. | |
| Score | mat6tube.com | Score rose 3 points to 58. | |
| Score | mastercard.com | Score fell 20 points to 58. | |
| Score | lowes.com | Score rose 7 points to 62. | |
| Availability | loopia.se | Came back online. | |
| Agent files | loopia.se | loopia.se: llms.txt added. | |
| Agent files | logitech.com | logitech.com: agents.md added. | |
| Score | logitech.com | Score rose 4 points to 81. | |
| Score | loggly.com | Score rose 7 points to 80. | |
| Score | lenovo.com | Score rose 7 points to 54. | |
| Score | legacy.com | Score fell 26 points to 66. | |
| Score | lefigaro.fr | Score rose 7 points to 63. | |
| Score | latimes.com | Score rose 3 points to 60. | |
| Score | lastpass.com | Score rose 7 points to 74. | |
| Availability | kuaishou.com | Went unreachable: UND_ERR_CONNECT_TIMEOUT. | |
| Crawler policy | kuaishou.com | kuaishou.com now allows ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, meta-externalagent. | |
| Score | kqbd.mobi | Score fell 3 points to 72. | |
| Score | ko-fi.com | Score fell 13 points to 40. | |
| Agent files | klarna.com | klarna.com: llms.txt removed. | |
| Score | klarna.com | Score rose 5 points to 78. | |
| Score | kit.com | Score fell 14 points to 78. | |
| Crawler policy | kicker.de | kicker.de now allows GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent. |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Changes in AI crawler policy." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.