Tranco rank 4,595
the-sun.com
the-sun.com scores 35 out of 100 for agent readiness. It blocks 8 of 11 answer-surface AI crawlers.
- Last measured
- 2026-08-09 18:21 UTC
- First seen
- 2026-08-09
- Rubric / probe
- v1.0.0 / v2.0.0
Closed to all crawlers
robots.txt disallows every crawler at the site root, so we read the policy and fetched no page. The access findings below stand; everything that would need the page itself was excluded rather than scored zero.
How this score is made up
Agent access8.2 / 38
- Answer-surface crawlers allowed. 8 of 11 blocked: ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent.
- Secondary crawlers allowed. 12 of 12 blocked: CCBot, Amazonbot, Bytespider, cohere-ai, MistralAI-User, YouBot, DuckAssistBot, kagi-fetcher, Diffbot, Google-NotebookLM, TavilyBot, FirecrawlAgent.
- Serves crawlers the same content. Not assessed. Our control request was challenged, so there is no clean baseline to compare against.
Machine-readable surface8 / 8
- robots.txt published. robots.txt is present and parseable.
- Sitemap declared in robots.txt. robots.txt points crawlers at a sitemap.
- llms.txt published. Not assessed. /llms.txt could not be fetched cleanly past the bot challenge.
- agents.md published. Not assessed. /agents.md could not be fetched cleanly past the bot challenge.
Content structurenot assessed
- Organization schema. Not assessed. Our control request was challenged by a bot wall.
- WebSite schema. Not assessed. Our control request was challenged by a bot wall.
- Additional structured data. Not assessed. Our control request was challenged by a bot wall.
- Readable without JavaScript. Not assessed. Our control request was challenged by a bot wall.
- Single top-level heading. Not assessed. Our control request was challenged by a bot wall.
- Semantic landmarks. Not assessed. Our control request was challenged by a bot wall.
Crawler policy
Read from the-sun.com/robots.txt. A crawler is listed as blocked when the rules deny it the site root. This operator names 3 AI crawlers explicitly, which means the policy is deliberate rather than inherited from a wildcard rule.
| Crawler | Operator | Tier | Named | Status |
|---|---|---|---|---|
| GPTBot | OpenAI | 1 | yes | Allowed |
| OAI-SearchBot | OpenAI | 1 | yes | Allowed |
| ChatGPT-User | OpenAI | 1 | yes | Allowed |
| ClaudeBot | Anthropic | 1 | no | Blocked |
| Claude-User | Anthropic | 1 | no | Blocked |
| Claude-SearchBot | Anthropic | 1 | no | Blocked |
| PerplexityBot | Perplexity | 1 | no | Blocked |
| Perplexity-User | Perplexity | 1 | no | Blocked |
| Google-Extended | 1 | no | Blocked | |
| Applebot-Extended | Apple | 1 | no | Blocked |
| meta-externalagent | Meta | 1 | no | Blocked |
| CCBot | Common Crawl | 2 | no | Blocked |
| Amazonbot | Amazon | 2 | no | Blocked |
| Bytespider | ByteDance | 2 | no | Blocked |
| cohere-ai | Cohere | 2 | no | Blocked |
| MistralAI-User | Mistral | 2 | no | Blocked |
| YouBot | You.com | 2 | no | Blocked |
| DuckAssistBot | DuckDuckGo | 2 | no | Blocked |
| kagi-fetcher | Kagi | 2 | no | Blocked |
| Diffbot | Diffbot | 2 | no | Blocked |
| Google-NotebookLM | 2 | no | Blocked | |
| TavilyBot | Tavily | 2 | no | Blocked |
| FirecrawlAgent | Firecrawl | 2 | no | Blocked |
Show this score
Free to embed. Always reflects the latest measurement, and links back here so anyone can check the working.
<a href="https://crawlindex.org/site/the-sun.com"><img src="https://crawlindex.org/badge/the-sun.com.svg" alt="CrawlIndex agent readiness score for the-sun.com" width="196" height="28"></a>Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "the-sun.com agent readiness." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.