26.24%
of the top 1000 websites name at least one AI crawler in robots.txt and disallow it, for the week of 2026-11-02.
A known bias in this number
sites behind anti-automation challenges are excluded and are plausibly more AI-hostile than average, so the panel likely under-represents AI-hostile sites.
See every domain in the panel, searchable.
Trend
- Targeted — named crawler, disallowed
- Effective — named or blanket disallow
Sub-series for this period
The headline figure counts sites that name at least one AI crawler in robots.txt and disallow it — a targeted block. It does not by itself include sites closed to every crawler by a blanket rule; the effective row below adds those in, so targeted and effective access are not the same number.
The coverage row is not a share of the web; it is the share of the panel this period's headline figure actually rests on. A site counts as conclusively observed if it served a usable robots.txt, or definitively answered that it has none, with a 404. Sites that timed out, refused the request, or sat behind an anti-automation challenge are excluded from both the numerator and the denominator of the headline — counted as neither compliant nor non-compliant.
Fixed series
| Series | Value |
|---|---|
| Blanket disallow (closed to every crawler) | 5.88% |
| Coverage | 98.70% |
| Effective (named or blanket) | 29.28% |
| Unreadable robots.txt | 0.60% |
| change_wow | 1.12% |
Per-agent series
| Series | Value |
|---|---|
| AI2Bot | 6.18% |
| AI2Bot-Dolma | 4.96% |
| Amazonbot | 11.85% |
| Applebot-Extended | 12.87% |
| Brightbot | 3.65% |
| Bytespider | 16.72% |
| CCBot | 18.03% |
| ChatGLM-Spider | 3.14% |
| ChatGPT-User | 11.55% |
| Claude-SearchBot | 9.63% |
| Claude-User | 9.02% |
| Claude-Web | 10.13% |
| ClaudeBot | 17.22% |
| DeepSeekBot | 4.46% |
| Devin | 2.63% |
| Diffbot | 11.65% |
| DuckAssistBot | 7.50% |
| GPTBot | 18.34% |
| Gemini-Deep-Research | 3.24% |
| Google-CloudVertexBot | 7.09% |
| Google-Extended | 15.91% |
| Google-NotebookLM | 2.94% |
| GoogleAgent-Mariner | 2.33% |
| ImagesiftBot | 7.19% |
| MistralAI-User | 5.78% |
| OAI-SearchBot | 10.74% |
| PanguBot | 5.47% |
| Perplexity-User | 8.31% |
| PerplexityBot | 14.59% |
| PhindBot | 2.53% |
| SBIntuitionsBot | 2.94% |
| TavilyBot | 2.43% |
| Timpibot | 8.00% |
| WrtnBot | 2.43% |
| YouBot | 8.61% |
| anthropic-ai | 10.94% |
| cohere-ai | 11.75% |
| cohere-training-data-crawler | 5.78% |
| iaskspider | 2.63% |
| meta-externalagent | 13.88% |
| meta-externalfetcher | 6.99% |
| meta-webindexer | 4.05% |
| omgili | 11.35% |
| omgilibot | 8.21% |
| webzio-extended | 5.98% |
Every published print
| Period | Targeted | Coverage | Status |
|---|---|---|---|
| 2026-11-02 | 26.24% | 98.70% | Final |
| 2026-10-26 | 25.13% | 99.10% | Final |
| 2026-10-19 | 24.87% | 99.30% | Final |
| 2026-10-12 | 24.30% | 99.60% | Final |
| 2026-10-05 | 24.32% | 99.50% | Final |
| 2026-09-28 | 24.10% | 99.60% | Final |
| 2026-09-14 | 23.97% | 99.70% | Final |
Restated periods
These periods were published, then revised. Nothing is deleted from the record: the earlier value is kept alongside the reason it changed.
| Period | Old value | New value | Reason |
|---|---|---|---|
| 2026-09-14 | 23.47% | 23.97% | methodology v2: agent set expanded from 16 to 45 tokens after an audit found v1 incomplete (Anthropic tracked 1 of 4 live tokens; Meta's fetcher, Cohere's training crawler and Mistral absent) |
Downloads
Every figure published on this site can be re-derived from these files. The two ledgers below are served as the exact bytes committed to the repository, not regenerated.
| File | Rows | Size | Description |
|---|---|---|---|
| Headline ledger | 24 | 3.1 KB | Every headline figure ever published, for both indices, including superseded vintages and the reason each was restated. Served as the exact bytes committed to the repository. |
| Series ledger | 1148 | 158.9 KB | Every sub-series: the per-crawler rates, effective and blanket blocking, coverage, the unreadable share and the like-for-like change. Same vintage rules as the headline ledger, same exact bytes. |
| Panel, 2026-11-02 | 1000 | 17.1 KB | The domains measured in this period, with their Tranco rank. This is the denominator of every figure published for the period. |
| Verdicts, 2026-11-02 | 44415 | 2.1 MB | One row per domain per tracked crawler: the stance that domain's robots.txt takes toward that token. The evidence behind every rate published for the period. |