Open data
Take the whole thing
The entire dataset is a handful of files you can download, diff and re-run. There is no API key, no rate limit and no signup, because a public measurement project that makes its data awkward to obtain is not really public.
Files
One JSON object per measured domain, sorted by domain. Contains the complete archived observation each score was computed from.
The full daily statistics series since the index began, including per-crawler, per-platform and per-network breakdowns.
Every recorded change in crawler policy, agent files, score or reachability.
When the last crawl ran, from where, and under which probe and rubric versions.
4,984 domains in the current file. Last crawl 2026-08-09 18:26 UTC from gha-ubuntu, probe v2.0.0, rubric v1.0.0.
Every past version, too
The dataset lives in a public git repository and is rewritten by the nightly crawl, so every state the index has ever been in is a commit. That makes any figure permanently checkable: if a report quotes a number from three months ago, you can check out that commit and recompute it.
The repository holds the code, the data and the history.
Recomputing a score yourself
Each record archives the observation, not just the verdict. Scoring is a pure function over that object, so running scoreObservation on a stored record reproduces the published score exactly. If it does not, that is a bug worth reporting.
Two fields that decide what a score means
score: null means the site could not be measured. It is not a zero and should be excluded from averages rather than coerced.
partial: true means some checks could not be observed, usually because the request met a bot wall, so the total was renormalised over the points that remained. A partial 80 and a complete 80 are not the same claim. Do not rank them together.
Licence and credit
The data is licensed CC BY 4.0. Use it in research, journalism or a commercial product. The only condition is credit to Fidget Labs BV.
CrawlIndex by Fidget Labs BV. "CrawlIndex dataset." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.