Skip to main content

Findings Leaderboard Crawlers Reports Badge Data Method

Search the index

Domain name

Search the index. A domain that is not here can still be measured live on the check page.

Search

Look up a domain

Searches the 3,714 domains in the nightly index. A domain that is not here can still be measured live.

Domain

CrawlIndex reads publicly served files. It fetches robots.txt, homepages and two well-known paths, exactly as any crawler would. It stores no personal data, sets no cookies, and runs no analytics or third-party tracking.

To leave the index, disallow CrawlIndexBot in your robots.txt. It is checked before anything else is requested and takes effect on the next crawl.

Methodology Glossary Coverage Dataset Add a domain Check a domain About API llms.txt Source

Research and data by Fidget Labs BV, Breda, Netherlands. Free to reuse under CC BY 4.0 with attribution.

Search the index | CrawlIndex