20.19%
Denominator: 421 conclusive domains. Coverage: 68.90% of the panel.
Provenance
This is why this page exists: the exact run that produced this figure, frozen alongside it rather than left to be inferred later.
| Vintage | 1 |
|---|---|
| Methodology | agents=2;protego=0.6.2 |
| Collector | 9654a8f |
| Computed at |
Every series value for this period
| Series | Value |
|---|---|
| Effective (named or blanket) | 23.75% |
| Blanket disallow (closed to every crawler) | 5.23% |
| Coverage | 68.90% |
| Unreadable robots.txt | 2.13% |
| Change since the previous reading | 1.94% |
| AI2Bot | 0.00% |
| AI2Bot-Dolma | 0.00% |
| Amazonbot | 1.66% |
| Applebot-Extended | 0.00% |
| Brightbot | 0.00% |
| Bytespider | 2.38% |
| CCBot | 8.79% |
| ChatGLM-Spider | 0.00% |
| ChatGPT-User | 3.80% |
| Claude-SearchBot | 0.00% |
| Claude-User | 0.00% |
| Claude-Web | 1.19% |
| ClaudeBot | 0.71% |
| DeepSeekBot | 0.00% |
| Devin | 0.00% |
| Diffbot | 0.00% |
| DuckAssistBot | 0.00% |
| GPTBot | 17.81% |
| Gemini-Deep-Research | 0.00% |
| Google-CloudVertexBot | 0.00% |
| Google-Extended | 8.55% |
| Google-NotebookLM | 0.00% |
| GoogleAgent-Mariner | 0.00% |
| ImagesiftBot | 0.24% |
| MistralAI-User | 0.00% |
| OAI-SearchBot | 0.00% |
| PanguBot | 0.00% |
| Perplexity-User | 0.00% |
| PerplexityBot | 1.19% |
| PhindBot | 0.00% |
| SBIntuitionsBot | 0.00% |
| TavilyBot | 0.00% |
| Timpibot | 0.00% |
| WrtnBot | 0.00% |
| YouBot | 0.48% |
| anthropic-ai | 2.85% |
| cohere-ai | 1.19% |
| cohere-training-data-crawler | 0.00% |
| iaskspider | 0.00% |
| meta-externalagent | 0.00% |
| meta-externalfetcher | 0.00% |
| meta-webindexer | 0.00% |
| omgili | 3.09% |
| omgilibot | 3.33% |
| webzio-extended | 0.00% |
Restatements
This period has never been restated: the value above is the only one ever published for it.
Suggested citation
Three years of AI-crawler blocking, 2024-02: 20.19% (vintage 1). https://allanderek.github.io/tallyhouse/agent-accessibility-history/releases/2024-02/index.html
The two ledgers below carry this period's rows among every other period's; anything named for this period alone is listed with them.
Downloads
Every figure published on this site can be re-derived from these files. The two ledgers below are served as the exact bytes committed to the repository, not regenerated.
| File | Rows | Size | Description |
|---|---|---|---|
| Headline ledger | 20 | 2.6 KB | Every headline figure ever published, for both indices, including superseded vintages and the reason each was restated. Served as the exact bytes committed to the repository. |
| Series ledger | 948 | 130.2 KB | Every sub-series: the per-crawler rates, effective and blanket blocking, coverage, the unreadable share and the like-for-like change. Same vintage rules as the headline ledger, same exact bytes. |
| Balanced panel | 611 | 12.5 KB | The domains measured by the historical index: those in the Tranco top 1000 at both ends of the span, with their rank at each end. Fixed across every period, which is what makes a change in that series a change in behaviour rather than in membership. |