What this measures
How many of the top 1000 websites tell AI crawlers to stay out?
Tallyhouse reads the robots.txt file published by each site in a fixed panel drawn from the Tranco top-sites list, and checks whether it names one of a fixed list of AI crawlers and disallows it. The headline figure is the share of the panel that does, published weekly.
Tranco is a research-grade ranking of the most popular domains, built by combining several underlying popularity sources and published daily. Each day's list carries a permanent identifier, so a specific list can be cited and retrieved later — which is what lets a reader check our panel against the exact list we drew it from, rather than take our word for it.
The full method behind that figure — how a verdict is reached, what counts as evidence, and the versioned changelog behind every published number — is set out on the methodology page.
Reproducibility
Every number this site publishes is read back from a committed, append-only ledger, never recomputed at render time. Cloning the repository and re-running the collector against the recorded panel reproduces every published figure exactly. When a value has to change, it is appended as a new vintage alongside a stated reason — the earlier value is never edited or deleted.
The ledgers themselves are published as CSV in the live index's downloads section.
How the panel is built
Panel N2P2W, captured 2026-09-17T15:47:47Z: first N domains of the Tranco list returning a conclusive robots.txt observation during the sweep.
556 of 1600 candidates examined were excluded, leaving a qualified panel of 1000 sites.
Related work
Tallyhouse is not the only project measuring this ground. The Agentic Web Index, published by Known Agents, tracks the same crawlers from the opposite side: not what a site's robots.txt says, but what a crawler actually does when it requests a page.
Known Agents draws on server-side traffic from more than 5,000 websites running its analytics, which lets it observe agent-versus-human traffic, bot spoofing, and, where Tallyhouse cannot follow, robots.txt compliance: whether a crawler honours the rule it was given. Tallyhouse reads the rule from outside; it has no way to see whether anyone obeys it. A reader who wants to know whether a crawler actually obeys a disallow, rather than whether one was published, should look there.
The two panels are built for different questions, and neither design is wrong for its own. Known Agents' sample is self-selected, drawn from sites that adopted its product, and it says so plainly: "Participating websites are not a random sample, and the qualifying set changes over time, so results show observed directional trends rather than a census of global web traffic." Tallyhouse's panel is fixed for the year and drawn from an external ranking with a citable list identifier, so a change in the headline number cannot be a change in who is being measured. That is a difference in design goal, not a defect in theirs: a daily operational index and a slow, reproducible one are built to answer different questions.
Known Agents publishes no raw data downloads; Tallyhouse publishes the collected robots.txt bodies, the ledger, and the code behind every figure, so a stranger can re-derive any number published here. Known Agents updates daily; Tallyhouse prints weekly and freezes each print. Together the two say more than either alone: they measure what crawlers do, we measure what sites ask for.