crawlindex

Search the index. A domain that is not here can still be measured live on the check page.

Tranco rank 4,270

livehindustan.com

livehindustan.com scores 83 out of 100 for agent readiness. It blocks 1 of 11 answer-surface AI crawlers. That is ahead of 93% of measured sites.

Score 83 out of 100, grade B

Agent Friendly

Number of measured sites in each ten-point score band
Score bandSites
0 to 90
10 to 198
20 to 2918
30 to 3953
40 to 49293
50 to 59669
60 to 69751
70 to 79477
80 to 89321
90 to 9923
Ahead of 93% of fully measured sites
Last measured
2026-11-22 10:21 UTC
First seen
2026-11-16
Rubric / probe
v2.0.0 / v3.0.0
VantageWhere the request came from. Origins serve differently by geography and IP reputation, so an observation is only comparable with another taken from the same place. Everything here is measured from GitHub runners in the US and EU. More
gha-ubuntu

Policy postureWhether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. More

Deliberate

robots.txt names AI crawlers by token. Somebody at this organisation decided what AI systems may do with the site.

Access archetypeThe shape of the policy rather than its size. Open, no training, assistant only, selective, walled, metered, or undeclared. More

No training

Training crawlers are blocked and the crawlers that answer live questions are not. This site wants to be cited without being absorbed.

Built on Next.js, served through Akamai. Compare against everything else on the same stack.

How this score is made up

Points earned in each score band
BandEarnedAvailableNominal maximum
Agent access42.34545
Machine-readable surface162525
Content structure253030

Agent access42.3 / 45

  • Answer-surface crawlers allowed. 1 of 11 blocked: meta-externalagent.
  • Secondary crawlers allowed. All 12 secondary crawlers are allowed.
  • Serves crawlers the same content. A crawler and a browser receive comparable responses.

Machine-readable surface16 / 25

  • robots.txt published. robots.txt is present and parseable.
  • Sitemap declared in robots.txt. robots.txt points crawlers at a sitemap.
  • llms.txt published. /llms.txt is served and its structure matches the spec.
  • agents.md published. No /agents.md.
  • Licence terms declared. No RSL License directive and no licence link relation. Reuse terms are undeclared.
  • Granular usage preferences declared. No Content-Signal directive. Policy is expressed only as allow or deny.
  • Agent card published. No agent card at /.well-known/agent-card.json.

Content structure25 / 30

  • Organization schema. Organization JSON-LD lets an agent resolve who publishes this site.
  • WebSite schema. WebSite JSON-LD present.
  • Additional structured data. Also declares: WebPage, MobileApplication, SiteNavigationElement, ItemList, BreadcrumbList.
  • Readable without JavaScript. 11,793 characters of text in the server response. Most crawlers do not execute JavaScript.
  • Single top-level heading. 1 h1 element found.
  • Semantic landmarks. Uses nav, header, footer.
  • Dateline declared. No machine-readable date. An agent cannot tell how current this page is.
  • Authorship declared. No declared author. An agent quoting this page has nobody to credit.

How this compares to its peers

A score out of 100 is not information on its own. Against the sites running the same platform and sitting behind the same edge network, it is: this site is at or ahead of every cohort it belongs to, which usually means the ceiling is the platform rather than the site. Cohorts under 25 measured sites are never published, so every comparison here is against a real group.

This site's score against the median of each group it belongs to
GroupMedian scoreSites in groupDifference
This site8310
Sites on Next.js66.532416.5
Sites behind Akamai6228521.0
The whole index62261321.0

What would move this score

The observable checks that did not earn full marks, heaviest first. Only these; a check we could not observe is not on the list, because it is not a fact about the site.

  1. +4agents.md published. No /agents.md.
  2. +3Dateline declared. No machine-readable date. An agent cannot tell how current this page is.
  3. +2.7Answer-surface crawlers allowed. 1 of 11 blocked: meta-externalagent.
  4. +2Licence terms declared. No RSL License directive and no licence link relation. Reuse terms are undeclared.
  5. +2Granular usage preferences declared. No Content-Signal directive. Policy is expressed only as allow or deny.
  6. +2Authorship declared. No declared author. An agent quoting this page has nobody to credit.
  7. +1Agent card published. No agent card at /.well-known/agent-card.json.

Extraction profile

How cheap this page is for a retrieval pipeline to chunk. Recorded and published, deliberately not scored: it describes a shape rather than a pass or a fail, and compressing it into the grade would destroy the useful part.

Server text
11,793 chars
Text density
1%
Subheadings
73
Lists
28
Tables
0
Feed
no

Crawler policy

Read from livehindustan.com/robots.txt. A crawler is listed as blocked when the rules deny it the site root. This operator names 2 AI crawlers explicitly, which means the policy is deliberate rather than inherited from a wildcard rule.

AI crawler access policy for livehindustan.com
CrawlerOperatorTierNamedStatus
GPTBotOpenAI1noAllowed
OAI-SearchBotOpenAI1noAllowed
ChatGPT-UserOpenAI1noAllowed
ClaudeBotAnthropic1noAllowed
Claude-UserAnthropic1noAllowed
Claude-SearchBotAnthropic1noAllowed
PerplexityBotPerplexity1noAllowed
Perplexity-UserPerplexity1noAllowed
Google-ExtendedGoogle1yesAllowed
Applebot-ExtendedApple1noAllowed
meta-externalagentMeta1yesBlocked
CCBotCommon Crawl2noAllowed
AmazonbotAmazon2noAllowed
BytespiderByteDance2noAllowed
cohere-aiCohere2noAllowed
MistralAI-UserMistral2noAllowed
YouBotYou.com2noAllowed
DuckAssistBotDuckDuckGo2noAllowed
kagi-fetcherKagi2noAllowed
DiffbotDiffbot2noAllowed
Google-NotebookLMGoogle2noAllowed
TavilyBotTavily2noAllowed
FirecrawlAgentFirecrawl2noAllowed

livehindustan.com has earned the Agent Friendly mark

Free to embed, no account and no fee. It regenerates from the nightly crawl, so it stays true, and it links back to this page so anyone can check the working in one click. What the mark means.

Shape

Circular. For a footer or an about page.

Theme

Follows the reader’s own light or dark setting. Note that this tracks the reader, not your page, so a dark-mode visitor sees the dark mark on a light site.

CrawlIndex agent readiness score for livehindustan.comOn a light page
On a dark page

The mark links back to this site's page for your domain, so anyone can check the claim in one click. It regenerates nightly, which means it stays true and it can change.

This measurement as JSON

crawl 2026-11-22 10:38 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,656 of 5,006 reachable

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "livehindustan.com agent readiness." https://crawlindex.org (measured 2026-11-22). Licensed CC BY 4.0.