24.32%
of the top 1000 websites name at least one AI crawler in robots.txt and disallow it, for the week of 2026-10-05.
A known bias in this number
sites behind anti-automation challenges are excluded and are plausibly more AI-hostile than average, so the panel likely under-represents AI-hostile sites.
See every domain in the panel, searchable.
Trend
- Targeted — named crawler, disallowed
- Effective — named or blanket disallow
Sub-series for this period
The headline figure counts sites that name at least one AI crawler in robots.txt and disallow it — a targeted block. It does not by itself include sites closed to every crawler by a blanket rule; the effective row below adds those in, so targeted and effective access are not the same number.
The coverage row is not a share of the web; it is the share of the panel this period's headline figure actually rests on. A site counts as conclusively observed if it served a usable robots.txt, or definitively answered that it has none, with a 404. Sites that timed out, refused the request, or sat behind an anti-automation challenge are excluded from both the numerator and the denominator of the headline — counted as neither compliant nor non-compliant.
Fixed series
| Series | Value |
|---|---|
| Blanket disallow (closed to every crawler) | 5.83% |
| Coverage | 99.50% |
| Effective (named or blanket) | 27.34% |
| Unreadable robots.txt | 0.20% |
| change_wow | 0.20% |
Per-agent series
| Series | Value |
|---|---|
| AI2Bot | 5.63% |
| AI2Bot-Dolma | 4.62% |
| Amazonbot | 9.75% |
| Applebot-Extended | 10.95% |
| Brightbot | 3.42% |
| Bytespider | 14.57% |
| CCBot | 16.38% |
| ChatGLM-Spider | 3.02% |
| ChatGPT-User | 11.16% |
| Claude-SearchBot | 8.54% |
| Claude-User | 8.04% |
| Claude-Web | 8.14% |
| ClaudeBot | 15.18% |
| DeepSeekBot | 4.22% |
| Devin | 2.51% |
| Diffbot | 9.65% |
| DuckAssistBot | 6.73% |
| GPTBot | 16.58% |
| Gemini-Deep-Research | 3.02% |
| Google-CloudVertexBot | 5.13% |
| Google-Extended | 14.17% |
| Google-NotebookLM | 2.61% |
| GoogleAgent-Mariner | 2.21% |
| ImagesiftBot | 6.83% |
| MistralAI-User | 5.23% |
| OAI-SearchBot | 10.05% |
| PanguBot | 4.92% |
| Perplexity-User | 7.34% |
| PerplexityBot | 14.07% |
| PhindBot | 2.31% |
| SBIntuitionsBot | 2.81% |
| TavilyBot | 2.21% |
| Timpibot | 7.54% |
| WrtnBot | 2.21% |
| YouBot | 7.94% |
| anthropic-ai | 9.35% |
| cohere-ai | 9.75% |
| cohere-training-data-crawler | 5.03% |
| iaskspider | 2.51% |
| meta-externalagent | 11.76% |
| meta-externalfetcher | 6.53% |
| meta-webindexer | 3.32% |
| omgili | 9.45% |
| omgilibot | 7.94% |
| webzio-extended | 5.73% |
Every published print
| Period | Targeted | Coverage | Status |
|---|---|---|---|
| 2026-10-05 | 24.32% | 99.50% | Final |
| 2026-09-28 | 24.10% | 99.60% | Final |
| 2026-09-14 | 23.97% | 99.70% | Final |
Restated periods
These periods were published, then revised. Nothing is deleted from the record: the earlier value is kept alongside the reason it changed.
| Period | Old value | New value | Reason |
|---|---|---|---|
| 2026-09-14 | 23.47% | 23.97% | methodology v2: agent set expanded from 16 to 45 tokens after an audit found v1 incomplete (Anthropic tracked 1 of 4 live tokens; Meta's fetcher, Cohere's training crawler and Mistral absent) |
Downloads
Every figure published on this site can be re-derived from these files. The two ledgers below are served as the exact bytes committed to the repository, not regenerated.
| File | Rows | Size | Description |
|---|---|---|---|
| Headline ledger | 20 | 2.6 KB | Every headline figure ever published, for both indices, including superseded vintages and the reason each was restated. Served as the exact bytes committed to the repository. |
| Series ledger | 948 | 130.2 KB | Every sub-series: the per-crawler rates, effective and blanket blocking, coverage, the unreadable share and the like-for-like change. Same vintage rules as the headline ledger, same exact bytes. |
| Panel, 2026-10-05 | 1000 | 17.1 KB | The domains measured in this period, with their Tranco rank. This is the denominator of every figure published for the period. |
| Verdicts, 2026-10-05 | 44775 | 2.1 MB | One row per domain per tracked crawler: the stance that domain's robots.txt takes toward that token. The evidence behind every rate published for the period. |