Reference
What everything here means
Every term used on a table, a chip or a score line, defined once. If a page shows you a word and does not explain it, that is a bug and it belongs here.
- Answer-surface crawler
- A crawler whose output reaches a person as an answer today: the ones behind ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and Meta AI. Blocking one has an immediate, visible cost.Tier 1 in the registry. Separated from the rest because blocking Common Crawl affects a future model, while blocking OAI-SearchBot removes you from an answer somebody is reading right now. Averaging the two would hide the difference.See it in the data
- Secondary crawler
- Dataset builders and smaller assistants. Common Crawl, Bytespider, Diffbot, retrieval APIs. Blocking these has a slower and more diffuse effect.See it in the data
- CrawlIndex Score
- Zero to one hundred, from three bands: whether AI crawlers are allowed at all (45 points), whether the site publishes machine-readable surfaces (25), and whether its content is structured enough to be read (30). Arithmetic over archived evidence. No model is involved.See it in the data
- Partial assessment
- Some checks could not be observed, usually because a bot wall answered instead of the site, so those points were removed from the total rather than failed. The remaining points are renormalised to one hundred. A partial score is not comparable with a complete one, which is why partial sites are kept out of ranked lists.See it in the data
- Stub response
- A homepage that answered with under 5,000 bytes and no readable text. That is an anti-automation placeholder served to our crawler, not the site. Everything derived from it is discarded rather than counted against the operator.See it in the data
- Cloaking
- Serving a crawler materially less than a browser gets, or refusing it outright, while robots.txt says nothing about it. Measured by fetching the same homepage twice, once as a browser and once as GPTBot, and comparing.See it in the data
- Policy gap
- robots.txt permits GPTBot and the server refuses GPTBot anyway. The operator published one policy and a different one is being enforced, almost always by an edge rule switched on above them.See it in the data
- Policy posture
- Whether anyone actually decided. Deliberate means robots.txt names AI crawlers by token. Inherited means it names none, so whatever AI policy exists is a side effect of generic rules. Blanket means one rule for everyone. Absent means no robots.txt at all.See it in the data
- Access archetype
- The shape of the policy rather than its size. Open, no training, assistant only, selective, walled, metered, or undeclared.See it in the data
- Edge network
- The CDN or reverse proxy in front of the origin: Cloudflare, Akamai, Fastly, CloudFront. It can block a crawler before the site ever sees the request, which is why blocking correlates better with the CDN than with anything the operator published.See it in the data
- Publishing platform
- What the site is built on: WordPress, Shopify, Next.js, Squarespace. Detected from artefacts of running it, never from a page merely mentioning the vendor by name.See it in the data
- llms.txt
- A markdown file at the site root that points an AI agent at the pages worth reading. A community convention rather than a ratified standard, and adoption is still small.See it in the data
- agents.md
- A root-level markdown file describing how agents should behave on the site. Also a convention rather than a standard.See it in the data
- Agent card
- A JSON document at /.well-known/agent-card.json advertising an agent-to-agent interface. Adoption across the most-visited domains is currently close to zero, which is why tracking the curve from the start is worth doing.See it in the data
- Content-Signal
- A robots.txt directive expressing granular preferences, for example search=yes, ai-train=no, use=reference. Cloudflare writes it into managed robots.txt, so its spread is partly a measure of Cloudflare rather than of publisher intent.See it in the data
- RSL licensing
- Really Simple Licensing. A License: line in robots.txt pointing at machine-readable reuse terms. A third answer to "may an AI read this" beyond yes and no.See it in the data
- Pay per crawl
- HTTP 402 Payment Required answered to an unpaid agent. Access is being metered and sold rather than refused, and it is tracked as its own category because averaging it with a block would misdescribe both.See it in the data
- Vantage
- Where the request came from. Origins serve differently by geography and IP reputation, so an observation is only comparable with another taken from the same place. Everything here is measured from GitHub runners in the US and EU.See it in the data
- Percentile
- The share of fully measured sites this one scores at least as well as. A score without a percentile has no referent.