crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Change feed

What moved recently

Robots.txt edits are never announced. This is every movement the crawler has detected, newest first. A domain's first measurement is a baseline and is not listed, and comparisons are never made across a probe change or a change of crawl location.

Recorded changes to AI crawler treatment
DateTypeDomainWhat changed
Scorezillow.comScore fell 28 points to 50.
Agent fileszalando.dezalando.de: llms.txt added.
Availabilityzalando.deCame back online.
Availabilityys7.comWent unreachable: homepage returned HTTP 404.
Scoreyouku.comScore fell 26 points to 52.
Scorexhzone.liveScore fell 12 points to 48.
Scorexhtv.liveScore fell 12 points to 48.
Scorexhstream.comScore fell 12 points to 48.
Scorexhspot.comScore fell 12 points to 48.
Scorexhsocial.comScore fell 12 points to 48.
Scorexhopen.comScore fell 12 points to 48.
Scorexhname.comScore fell 12 points to 48.
Scorexhlink.blogScore fell 12 points to 48.
Scorexhfuck.comScore fell 12 points to 48.
Scorexhfanclub.comScore fell 12 points to 48.
Scorexhco.liveScore fell 12 points to 48.
Scorexhamster46.desiScore fell 12 points to 48.
Scorexhamster2.comScore fell 12 points to 48.
Scorexhamster19.comScore fell 12 points to 48.
Scorexhamster1.desiScore fell 12 points to 48.
Scorexhamster.desiScore fell 12 points to 48.
Scorexhamster.comScore fell 12 points to 48.
Scorexhaccess.comScore fell 12 points to 48.
Scorevivo.comScore fell 7 points to 49.
Availabilityvisa.cnWent unreachable: ETIMEDOUT.
Scorevigo.techScore rose 7 points to 64.
Scoreunruly.coScore fell 7 points to 66.
Availabilityunhcr.orgCame back online.
Scoreumeng.comScore rose 6 points to 80.
Scoretumblr.comScore rose 4 points to 54.
Scoretrulia.comScore fell 23 points to 69.
Availabilitytokyomotion.netCame back online.
Scoretime.comScore fell 6 points to 67.
Agent filestime.comtime.com: llms.txt removed.
Scoretilda.wsScore fell 23 points to 55.
Availabilitythisvid.comWent unreachable: homepage returned HTTP 502.
Scorethestar.comScore rose 3 points to 34.
Scoresxyprn.comScore rose 29 points to 84.
Scorestbid.ruScore rose 72 points to 78.
Crawler policystbid.rustbid.ru now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
Scoresplunkcloud.comScore fell 13 points to 65.
Scoreskyscanner.netScore fell 41 points to 45.
Scoresexwabs.comScore fell 7 points to 56.
Scoreserpro.gov.brScore fell 3 points to 58.
Scoresenderscore.comScore rose 19 points to 92.
Availabilitysedo.comCame back online.
Scoresamsclub.comScore fell 25 points to 67.
Scorerockstargames.comScore rose 21 points to 59.
Crawler policyprimevideo.comprimevideo.com now allows ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, meta-externalagent.
Availabilityprimevideo.comWent unreachable: homepage returned HTTP 503.
Scorepriceline.comScore fell 21 points to 63.
Availabilitypornpics.comWent unreachable: ECONNRESET.
Scoreplos.orgScore rose 14 points to 92.
Scoreohmy.bidScore rose 7 points to 59.
Availabilitymyoffice.ruWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Scoremuni.czScore fell 7 points to 59.
Availabilitymovieswood.cloudWent unreachable: homepage returned HTTP 523.
Scoremintegral.netScore fell 7 points to 55.
Scoremintegral.comScore fell 7 points to 55.
Scoremimecast.comScore fell 7 points to 67.
Scoremilfporn.tvScore rose 4 points to 78.
Availabilitymileroticos.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Availabilitymikrotik.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Crawler policykuaishou.comkuaishou.com now blocks ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, meta-externalagent.
Availabilitykuaishou.comCame back online.
Scoreitch.ioScore fell 27 points to 65.
Scoreipcc.chScore rose 6 points to 84.
Scoreintuit.comScore rose 6 points to 92.
Scoreindeed.comScore fell 6 points to 78.
Scoreign.comScore rose 3 points to 29.
Scorehrw.orgScore fell 21 points to 63.
Availabilityhinet.netCame back online.
Availabilityhawaii.eduWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Scoregotinder.comScore rose 10 points to 58.
Availabilitygooglesource.comWent unreachable: homepage returned HTTP 503.
Availabilitygoneo.deCame back online.
Scoregmt.ioScore rose 9 points to 86.
Agent filesgmt.iogmt.io: llms.txt added.
Scorefreebsd.orgScore fell 3 points to 57.
Scorefindagrave.comScore rose 5 points to 19.
Scorefederalregister.govScore rose 7 points to 62.
Scorefaphouse2.comScore rose 20 points to 78.
Scorefaphouse.comScore rose 20 points to 78.
Scorefamilysearch.orgScore fell 3 points to 58.
Scoreexpress.co.ukScore fell 17 points to 61.
Crawler policyexpress.co.ukexpress.co.uk now blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot-Extended.
Scoreerome.comScore rose 3 points to 55.
Scoreeporner.comScore fell 8 points to 59.
Scoreeenadu.netScore fell 28 points to 42.
Crawler policyeenadu.neteenadu.net now blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
Scoreecfr.govScore rose 10 points to 62.
Scoredominos.comScore rose 3 points to 64.
Scorecookiebot.comScore fell 7 points to 81.
Availabilityconsultant.ruCame back online.
Scorecolostate.eduScore rose 7 points to 77.
Scorecheckpoint.comScore fell 7 points to 70.
Scorechaturbate.comScore fell 14 points to 78.
Scorecapitaloneshopping.comScore fell 3 points to 72.
Scorebu.eduScore rose 3 points to 69.
Availabilitybrivo.comCame back online.
Scorebritishairways.comScore rose 7 points to 70.
Scorebmw-motorrad.com.cnScore fell 7 points to 70.
Scorebiorxiv.orgScore rose 40 points to 78.
Crawler policybiorxiv.orgbiorxiv.org now allows GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot.
Scoreav380.netScore fell 36 points to 48.
Agent filesarlo.comarlo.com: llms.txt removed.
Scoreappfolio.comScore fell 10 points to 71.
Scoreamocrm.ruScore rose 9 points to 84.
Agent filesamocrm.ruamocrm.ru: llms.txt added.
Scorealamy.comScore fell 10 points to 62.
Scoreafrinic.netScore rose 7 points to 77.
Availabilityafraid.orgCame back online.
Availabilityadotmob.comCame back online.
Scorea.coScore rose 33 points to 78.
Score1688.comScore fell 36 points to 48.
Scorezomato.comScore fell 26 points to 52.
Scorezhihu.comScore fell 7 points to 42.
Availabilityzalando.deWent unreachable: timeout.
Agent fileszalando.dezalando.de: llms.txt removed.
Scoreyumpu.comScore rose 9 points to 54.
Scoreyr.noScore fell 7 points to 31.
Availabilityyieldmo.comWent unreachable: UND_ERR_SOCKET.
Scoreyandex.netScore rose 3 points to 52.
Scoreyandex.byScore fell 3 points to 56.
Scorexiaomi.netScore rose 7 points to 60.
Scorexiaomi.comScore rose 7 points to 60.
Scorexerox.comScore fell 7 points to 56.
Scorexero.comScore rose 7 points to 73.
Scorewelt.deScore rose 7 points to 50.
Scoreweforum.orgScore rose 7 points to 78.
Crawler policywayground.comwayground.com now allows ClaudeBot.
Scorewashingtontimes.comScore fell 11 points to 78.
Scorewallapop.comScore rose 24 points to 78.
Scorewalgreens.comScore rose 7 points to 64.
Scorevoanews.comScore rose 7 points to 67.
Scorevnexpress.netScore fell 4 points to 70.
Availabilityvkvideo.ruCame back online.
Scorevivox.comScore rose 7 points to 67.
Scorevivo.comScore rose 7 points to 56.
Availabilityvisa.cnCame back online.
Availabilityvirtualbox.orgCame back online.
Scorevigo.techScore fell 7 points to 57.
Scoreva.govScore rose 13 points to 76.
Scoreusaa.comScore rose 4 points to 48.
Scoreunity3d.comScore rose 7 points to 79.
Scoreunity.comScore rose 7 points to 79.
Availabilityunhcr.orgWent unreachable: ECONNRESET.
Scoreunfccc.intScore fell 27 points to 57.
Scoreunep.orgScore rose 4 points to 65.
Scoreun.orgScore fell 3 points to 45.
Scoreumeng.comScore fell 6 points to 74.
Crawler policyumd.eduumd.edu now blocks GPTBot, PerplexityBot.
Scoreumd.eduScore fell 14 points to 64.
Scoreukdevilz.comScore rose 6 points to 84.
Scoretumblr.comScore fell 4 points to 50.
Scoretrusteer.comScore rose 4 points to 58.
Scoretrendmicro.comScore rose 7 points to 77.
Scoretreasury.govScore rose 7 points to 57.
Agent filestraveloka.comtraveloka.com: llms.txt removed.
Scoretraveloka.comScore rose 7 points to 78.
Scoretorob.comScore fell 3 points to 73.
Availabilitytokyomotion.netWent unreachable: timeout.
Scoretoday.comScore rose 7 points to 32.
Scoretizen.orgScore rose 14 points to 78.
Agent filestime.comtime.com: llms.txt added.
Scoretime.comScore rose 6 points to 73.
Scoretilda.wsScore rose 23 points to 78.
Agent filestheweathernetwork.comtheweathernetwork.com: llms.txt removed.
Scoretheweathernetwork.comScore rose 4 points to 78.
Scorethestar.comScore rose 17 points to 31.
Availabilitythemegrill.comWent unreachable: homepage returned HTTP 520.
Scoreterra.com.brScore rose 7 points to 77.
Scoretelekom.deScore rose 11 points to 78.
Scoretelegraph.co.ukScore rose 5 points to 37.
Agent filesteepublic.comteepublic.com: llms.txt added.
Scoresyosetu.comScore fell 26 points to 58.
Scoresuperhosting.bgScore fell 23 points to 55.
Scoresun.comScore rose 7 points to 60.
Agent filessubstack.comsubstack.com: llms.txt removed.
Scoresubstack.comScore rose 3 points to 78.
Scoresubito.itScore rose 7 points to 61.
Scorestudentaid.govScore rose 4 points to 45.
Availabilitystanford.eduCame back online.
Scoresportskeeda.comScore rose 11 points to 78.
Scoresotwe.comScore rose 12 points to 90.
Availabilitysonobi.comWent unreachable: homepage returned HTTP 502.
Scoresmithsonianmag.comScore fell 12 points to 66.
Availabilitysmartsheet.comCame back online.
Agent filessmartsheet.comsmartsheet.com: llms.txt added.
Availabilitysitescout.comCame back online.
Agent filessiteground.bizsiteground.biz: llms.txt added.
Scoresiteground.bizScore rose 14 points to 92.
Availabilitysinoptik.uaCame back online.
Scoresignifyd.comScore rose 7 points to 89.
Scoresicredi.com.brScore rose 7 points to 59.
Scoresi.eduScore fell 6 points to 78.
Crawler policysex-studentki.vodkasex-studentki.vodka now blocks GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Google-Extended, meta-externalagent.
Scoresex-studentki.vodkaScore fell 29 points to 49.
Crawler policysex-studentki.teamsex-studentki.team now blocks GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Google-Extended, meta-externalagent.
Scoresex-studentki.teamScore fell 29 points to 49.
Scoresephora.comScore rose 7 points to 68.
Crawler policyseattletimes.comseattletimes.com now allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
Scoreseattletimes.comScore rose 44 points to 78.
Scoreschwab.comScore rose 7 points to 64.
Scoreschoology.comScore fell 12 points to 66.
Availabilitysbis.ruCame back online.
Scoresbb.chScore rose 21 points to 78.
Scoresatta-king-fast.comScore fell 30 points to 48.
Scoresalesforce.comScore rose 7 points to 77.
Availabilitysaby.ruCame back online.
Availabilityruliweb.comCame back online.
Scorerule34.xxxScore rose 6 points to 84.
Availabilityrosminzdrav.ruWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Scorerockstargames.comScore fell 21 points to 38.
Availabilityrevolve.comWent unreachable: timeout.
Scorerentalcars.comScore rose 11 points to 78.
Scoreredwap.sexScore rose 33 points to 84.
Scoreredcross.orgScore rose 7 points to 54.
Scorerecruit.co.jpScore rose 8 points to 78.
Availabilityrajasthan.gov.inCame back online.
Agent filesrajasthan.gov.inrajasthan.gov.in: llms.txt added.
Scoreradiofrance.frScore fell 7 points to 39.
Availabilityquago.ioWent unreachable: UND_ERR_SOCKET.
Scorepussyspace.comScore fell 17 points to 36.
Availabilityproperty24.comCame back online.
Scoreprnewswire.comScore fell 25 points to 67.
Availabilityprivacymanager.ioWent unreachable: ENOTFOUND.
Agent filesprivacymanager.ioprivacymanager.io: llms.txt removed.
Scorepremierleague.comScore fell 14 points to 64.
Scorepravda.com.uaScore fell 21 points to 71.
Scoreposhmark.comScore fell 21 points to 63.
Scoreplex.tvScore fell 3 points to 57.
Availabilitypku.edu.cnWent unreachable: ERR_TLS_CERT_ALTNAME_INVALID.
Scorepin.itScore rose 5 points to 53.
Scorepimpbunny.comScore fell 12 points to 29.
Scorepikabu.ruScore fell 25 points to 67.
Scoreperplexity.aiScore fell 47 points to 45.
Scorepearson.comScore rose 7 points to 61.
Availabilitypcre.orgWent unreachable: UND_ERR_SOCKET.
Scorepchome.com.twScore fell 10 points to 68.
Scorepatreon.comScore rose 37 points to 90.
Scorepardot.comScore rose 7 points to 60.
Availabilityover-blog.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Scoreosu.eduScore rose 14 points to 78.
Scoreopenai.comScore rose 7 points to 64.
Scoreondemand.comScore rose 7 points to 60.
Scoreohmy.bidScore fell 7 points to 52.
Scoreogury.ioScore rose 7 points to 80.
Scorenyu.eduScore rose 24 points to 78.
Scorenyc.govScore rose 7 points to 63.
Scorenrk.noScore fell 7 points to 28.
Scorenpr.orgScore rose 3 points to 60.
Availabilitynovell.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Scorenokia.comScore rose 7 points to 67.
Scorenjavtv.comScore fell 23 points to 55.
Scorenike.comScore rose 7 points to 77.
Scorenext.co.ukScore rose 7 points to 69.
Scorenespresso.comScore rose 3 points to 64.
Scorenbcnews.comScore rose 7 points to 30.
Scorenbc.comScore fell 5 points to 61.
Scorenaukri.comScore rose 7 points to 45.
Scoremyportfolio.comScore rose 30 points to 78.
Scoremsu.eduScore fell 19 points to 73.
Scoremsdn.comScore rose 7 points to 64.
Scoremoodle.comScore rose 5 points to 78.
Availabilitymobilityware.comWent unreachable: UND_ERR_SOCKET.
Scoremiwifi.comScore rose 29 points to 78.
Scoremissav123.comScore fell 23 points to 55.
Scoremissav.wsScore fell 23 points to 55.
Scoremissav.liveScore fell 23 points to 55.
Scoremissav.aiScore fell 23 points to 55.
Scoremilfporn.tvScore fell 4 points to 74.
Scoremicrosoft.comScore rose 4 points to 65.
Scoremichaels.comScore rose 14 points to 61.
Scoremi.comScore rose 7 points to 60.
Scoremedicare.govScore rose 7 points to 63.
Agent filesmediafire.commediafire.com: llms.txt added.
Scoremedal.tvScore rose 7 points to 71.
Scoremax.comScore fell 7 points to 60.
Scoremat6tube.comScore rose 3 points to 58.
Scoremastercard.comScore fell 20 points to 58.
Scorelowes.comScore rose 7 points to 62.
Availabilityloopia.seCame back online.
Agent filesloopia.seloopia.se: llms.txt added.
Agent fileslogitech.comlogitech.com: agents.md added.
Scorelogitech.comScore rose 4 points to 81.
Scoreloggly.comScore rose 7 points to 80.
Scorelenovo.comScore rose 7 points to 54.
Scorelegacy.comScore fell 26 points to 66.
Scorelefigaro.frScore rose 7 points to 63.
Scorelatimes.comScore rose 3 points to 60.
Scorelastpass.comScore rose 7 points to 74.
Availabilitykuaishou.comWent unreachable: UND_ERR_CONNECT_TIMEOUT.
Crawler policykuaishou.comkuaishou.com now allows ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, meta-externalagent.
Scorekqbd.mobiScore fell 3 points to 72.
Scoreko-fi.comScore fell 13 points to 40.
Agent filesklarna.comklarna.com: llms.txt removed.
Scoreklarna.comScore rose 5 points to 78.
Scorekit.comScore fell 14 points to 78.
Crawler policykicker.dekicker.de now allows GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Changes in AI crawler policy." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.