What is measured
How many of the top 1000 websites tell AI crawlers to stay out?
The measure behind that question is the share of a fixed panel whose robots.txt names a tracked AI crawler and disallows it. Naming and disallowing is the test, not mere closure: a site shut to every crawler by a blanket rule that names none of them is not counted in this headline. That is not hidden — it is published separately, as its own blanket figure, in the index page's series.
How a verdict is reached
A token the file never names is Unmentioned — not consent, since a blanket rule naming nobody could still be catching it. Otherwise the token's own group of rules is evaluated: if every probe path is disallowed the verdict is a full block; if the group disallows some paths but not others it is a partial block; otherwise the token is allowed.
That evaluation is done by the pinned protego parser rather than a hand-rolled one, so a rule is read the way a conformant robots.txt library reads it. The probe paths themselves are fixed, because changing them would change published numbers:
//index.html/about/news/article-1/api/v1/data/images/photo.jpg
What counts as evidence
Only two outcomes tell us a site's policy: Fetched and NoRobotsTxt. A server can also answer and still withhold the file — Challenged and ServerError — and those cases are excluded from the rate rather than assumed permissive, published as their own series so the blind spot stays visible rather than being quietly absorbed into either side of the headline.
Collection also caps how much of any one file it will read: a body larger than 512.0 KB is not read in full.
Cadence and the collection window
A period is a week, identified by its Monday. Collection stays open for 72 hours after a period begins, during which re-collection can still improve coverage; once that window closes the figure is frozen.
A print whose coverage falls below 97% is marked provisional, since more evidence could still arrive for it inside the window.
The crawler
Every observation this method rests on is made by one crawler, identifying itself as TallyhouseIndexBot/1.0 (+https://allanderek.github.io/tallyhouse/about/crawler/). More on what it fetches and how to verify it.
Methodology versions
A methodology version is the tracked agent set combined with the pinned parser: anything that can change a verdict from identical raw evidence. It is composed from those two rather than written by hand, so neither can be bumped without the ledger noticing.
| Version | Adopted | Summary | What changed | Why | In force for | Restated |
|---|---|---|---|---|---|---|
agents=1;protego=0.6.2 | 2026-09-17 | First published methodology. | Initial agent set of 16 tokens, classified with protego 0.6.2 against a frozen 1,000-domain panel. | The starting point, published so that later changes have something to be measured against. | not currently in force | Produced 2026-09-14; since restated under a later version. |
agents=2;protego=0.6.2 | 2026-09-19 | Agent set expanded from 16 tokens to 45. | Added 29 tokens: operator-run crawlers that were live and widely named in robots.txt but untracked. Generic scraping tooling was deliberately not added, since a site naming it is not stating a position on AI crawling. | An audit of tokens appearing in the collected files found the original set incomplete enough to understate the headline: Anthropic was tracked by one of four live tokens, and Meta's fetcher, Cohere's training crawler and Mistral were absent entirely. Every affected published figure was restated as a new vintage with this reason recorded, rather than edited. | 19 periods, 2023-01 to 2026-10-05 | — |
For full detail, see the design documents in the repository.