Tranco rank 4,618
xda-developers.com
xda-developers.com scores 54 out of 100 for agent readiness. It blocks 4 of 11 answer-surface AI crawlers.
Score 54 out of 100, grade D
- Last measured
- 2026-08-09 18:24 UTC
- First seen
- 2026-08-09
- Rubric / probe
- v1.0.0 / v2.0.0
Built on WordPress. Compare against everything else on the same stack.
How this score is made up
Agent access30.8 / 45
- Answer-surface crawlers allowed. 4 of 11 blocked: GPTBot, ChatGPT-User, ClaudeBot, Applebot-Extended.
- Secondary crawlers allowed. 5 of 12 blocked: CCBot, Bytespider, cohere-ai, YouBot, Diffbot.
- Serves crawlers the same content. A crawler and a browser receive comparable responses.
Machine-readable surface8 / 25
- robots.txt published. robots.txt is present and parseable.
- Sitemap declared in robots.txt. robots.txt points crawlers at a sitemap.
- llms.txt published. No /llms.txt.
- agents.md published. No /agents.md.
Content structure15 / 30
- Organization schema. No Organization JSON-LD. Agents cannot reliably attribute this site to an entity.
- WebSite schema. WebSite JSON-LD present.
- Additional structured data. No further JSON-LD types on the homepage.
- Readable without JavaScript. 9,016 characters of text in the server response. Most crawlers do not execute JavaScript.
- Single top-level heading. 0 h1 elements found.
- Semantic landmarks. Uses main, nav, header, footer, article, aside.
Crawler policy
Read from xda-developers.com/robots.txt. A crawler is listed as blocked when the rules deny it the site root. This operator names 9 AI crawlers explicitly, which means the policy is deliberate rather than inherited from a wildcard rule.
| Crawler | Operator | Tier | Named | Status |
|---|---|---|---|---|
| GPTBot | OpenAI | 1 | yes | Blocked |
| OAI-SearchBot | OpenAI | 1 | no | Allowed |
| ChatGPT-User | OpenAI | 1 | yes | Blocked |
| ClaudeBot | Anthropic | 1 | yes | Blocked |
| Claude-User | Anthropic | 1 | no | Allowed |
| Claude-SearchBot | Anthropic | 1 | no | Allowed |
| PerplexityBot | Perplexity | 1 | no | Allowed |
| Perplexity-User | Perplexity | 1 | no | Allowed |
| Google-Extended | 1 | no | Allowed | |
| Applebot-Extended | Apple | 1 | yes | Blocked |
| meta-externalagent | Meta | 1 | no | Allowed |
| CCBot | Common Crawl | 2 | yes | Blocked |
| Amazonbot | Amazon | 2 | no | Allowed |
| Bytespider | ByteDance | 2 | yes | Blocked |
| cohere-ai | Cohere | 2 | yes | Blocked |
| MistralAI-User | Mistral | 2 | no | Allowed |
| YouBot | You.com | 2 | yes | Blocked |
| DuckAssistBot | DuckDuckGo | 2 | no | Allowed |
| kagi-fetcher | Kagi | 2 | no | Allowed |
| Diffbot | Diffbot | 2 | yes | Blocked |
| Google-NotebookLM | 2 | no | Allowed | |
| TavilyBot | Tavily | 2 | no | Allowed |
| FirecrawlAgent | Firecrawl | 2 | no | Allowed |
Show this score
Free to embed. Always reflects the latest measurement, and links back here so anyone can check the working.
<a href="https://crawlindex.org/site/xda-developers.com"><img src="https://crawlindex.org/badge/xda-developers.com.svg" alt="CrawlIndex agent readiness score for xda-developers.com" width="196" height="28"></a>Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "xda-developers.com agent readiness." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.