Tranco rank 5,009
infura-ipfs.io
infura-ipfs.io scores 6 out of 100 for agent readiness. It blocks 11 of 11 answer-surface AI crawlers.
Measured
- Last measured
- 2026-08-17 03:41 UTC
- First seen
- 2026-08-17
- Rubric / probe
- v2.0.0 / v3.0.0
Vantage
Where the request came from. Origins serve differently by geography and IP reputation, so an observation is only comparable with another taken from the same place. Everything here is measured from GitHub runners in the US and EU. More- gha-ubuntu
Policy posture
Whether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. More
Policy posture
Whether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all. MoreBlanket
One rule for every crawler, allow nothing. This is a decision, but it is not a decision about AI specifically.
Access archetype
The shape of the policy rather than its size. Open, no training, assistant only, selective, walled, metered, or undeclared. More
Access archetype
The shape of the policy rather than its size. Open, no training, assistant only, selective, walled, metered, or undeclared. MoreWalled
Every answer-surface crawler is blocked. This site is invisible to AI answers by choice.
Closed to all crawlers
robots.txt disallows every crawler at the site root, so we read the policy and fetched no page. The access findings below stand; everything that would need the page itself was excluded rather than scored zero.
How this score is made up
| Band | Earned | Available | Nominal maximum |
|---|---|---|---|
| Agent access | 0 | 38 | 45 |
| Machine-readable surface | 3 | 11 | 25 |
| Content structure | 0 | 0 | 30 |
Agent access0 / 38
- Answer-surface crawlers allowed. 11 of 11 blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- Secondary crawlers allowed. 12 of 12 blocked: CCBot, Amazonbot, Bytespider, cohere-ai, MistralAI-User, YouBot, DuckAssistBot, kagi-fetcher, Diffbot, Google-NotebookLM, TavilyBot, FirecrawlAgent.
- Serves crawlers the same content. Not assessed. There is no clean baseline to compare against, because our control request was challenged by a bot wall.
Machine-readable surface3 / 11
- robots.txt published. robots.txt is present and parseable.
- Sitemap declared in robots.txt. robots.txt does not declare a sitemap.
- llms.txt published. Not assessed. /llms.txt could not be read cleanly, because our control request was challenged by a bot wall.
- agents.md published. Not assessed. /agents.md could not be read cleanly, because our control request was challenged by a bot wall.
- Licence terms declared. No RSL License directive and no licence link relation. Reuse terms are undeclared.
- Granular usage preferences declared. No Content-Signal directive. Policy is expressed only as allow or deny.
- Agent card published. Not assessed. /.well-known/agent-card.json could not be read cleanly, because our control request was challenged by a bot wall.
Content structurenot assessed
- Organization schema. Not assessed, because our control request was challenged by a bot wall.
- WebSite schema. Not assessed, because our control request was challenged by a bot wall.
- Additional structured data. Not assessed, because our control request was challenged by a bot wall.
- Readable without JavaScript. Not assessed, because our control request was challenged by a bot wall.
- Single top-level heading. Not assessed, because our control request was challenged by a bot wall.
- Semantic landmarks. Not assessed, because our control request was challenged by a bot wall.
- Dateline declared. Not assessed, because our control request was challenged by a bot wall.
- Authorship declared. Not assessed, because our control request was challenged by a bot wall.
What would move this score
The observable checks that did not earn full marks, heaviest first. Only these; a check we could not observe is not on the list, because it is not a fact about the site.
- +30Answer-surface crawlers allowed. 11 of 11 blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- +8Secondary crawlers allowed. 12 of 12 blocked: CCBot, Amazonbot, Bytespider, cohere-ai, MistralAI-User, YouBot, DuckAssistBot, kagi-fetcher, Diffbot, Google-NotebookLM, TavilyBot, FirecrawlAgent.
- +4Sitemap declared in robots.txt. robots.txt does not declare a sitemap.
- +2Licence terms declared. No RSL License directive and no licence link relation. Reuse terms are undeclared.
- +2Granular usage preferences declared. No Content-Signal directive. Policy is expressed only as allow or deny.
Extraction profile
How cheap this page is for a retrieval pipeline to chunk. Recorded and published, deliberately not scored: it describes a shape rather than a pass or a fail, and compressing it into the grade would destroy the useful part.
- Server text
- 0 chars
- Text density
- 0%
- Subheadings
- 0
- Lists
- 0
- Tables
- 0
- Feed
- no
Crawler policy
Read from infura-ipfs.io/robots.txt. A crawler is listed as blocked when the rules deny it the site root. This operator names no AI crawler explicitly.
| Crawler | Operator | Tier | Named | Status |
|---|---|---|---|---|
| GPTBot | OpenAI | 1 | no | Blocked |
| OAI-SearchBot | OpenAI | 1 | no | Blocked |
| ChatGPT-User | OpenAI | 1 | no | Blocked |
| ClaudeBot | Anthropic | 1 | no | Blocked |
| Claude-User | Anthropic | 1 | no | Blocked |
| Claude-SearchBot | Anthropic | 1 | no | Blocked |
| PerplexityBot | Perplexity | 1 | no | Blocked |
| Perplexity-User | Perplexity | 1 | no | Blocked |
| Google-Extended | 1 | no | Blocked | |
| Applebot-Extended | Apple | 1 | no | Blocked |
| meta-externalagent | Meta | 1 | no | Blocked |
| CCBot | Common Crawl | 2 | no | Blocked |
| Amazonbot | Amazon | 2 | no | Blocked |
| Bytespider | ByteDance | 2 | no | Blocked |
| cohere-ai | Cohere | 2 | no | Blocked |
| MistralAI-User | Mistral | 2 | no | Blocked |
| YouBot | You.com | 2 | no | Blocked |
| DuckAssistBot | DuckDuckGo | 2 | no | Blocked |
| kagi-fetcher | Kagi | 2 | no | Blocked |
| Diffbot | Diffbot | 2 | no | Blocked |
| Google-NotebookLM | 2 | no | Blocked | |
| TavilyBot | Tavily | 2 | no | Blocked |
| FirecrawlAgent | Firecrawl | 2 | no | Blocked |
No mark at this score
The embeddable mark starts at grade B, because offering a graphic nobody would put on their own site is a pretence rather than a feature. The list above is the useful version: for most sites the points are concentrated in two or three changes. How the mark works.
The neutral mark above exists and is free to use if you want to show that the site is independently measured whatever the number says.
crawl 2026-08-17 04:00 UTCprobe 3.0.0rubric 2.0.0registry 1.0.0vantage gha-ubuntu3,651 of 5,006 reachable
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "infura-ipfs.io agent readiness." https://crawlindex.org (measured 2026-08-17). Licensed CC BY 4.0.