Three years of AI-crawler blocking

How it is read

This index reads the robots.txt that Common Crawl captured at the time, rather than anything our own crawler fetched, so it can reach back to before this project existed.

It is parsed by the same parser as the live index, so the two indices differ in how evidence is gathered and not at all in how it is read.

The panel

611 domains: those present in the Tranco top 1000 at both endpoints, K2K4W (2023-02-01) and N2P2W (2026-09-16).

Membership is fixed across the span: 389 of 1000 early entries absent at the later endpoint.

Members were prominent at both endpoints, so the panel is biased toward durably significant sites; sites that rose or fell within the span are absent.

Coverage

Common Crawl did not capture a robots.txt for every panel domain in every crawl, so each period publishes the share it could read. Coverage ranges from 61.70% to 68.90% across the published periods.

Domains it could not read are excluded from the rate rather than assumed permissive.

Why these figures are final

The live index marks a print provisional when coverage is low, because more evidence can still arrive inside its collection window. Common Crawl's archive is closed, so a historical reading can never be restated on coverage grounds and is final the moment it is computed. Coverage is published per period instead.

Which crawls

Roughly quarterly from January 2023 to August 2026. Quarterly rather than every crawl because the question is the shape of the rise — a step at GPTBot's launch in August 2023, or a steady climb — and a quarter of the requests answers it. Denser coverage can be added later; the collector is resumable and skips crawls already gathered.