Purpose: Training. Bulk crawling to build model training data.
The Allen Institute's crawler. A non-profit research institute that publishes both its models and its training corpora openly.
5.63%
of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.
How sites treat this token
| Stance | Domains | What it means |
|---|---|---|
| FullBlock | 52 | Every path is disallowed for this token. |
| PartialBlock | 4 | Some paths are disallowed for this token; others are not. |
| Allowed | 2 | The token is named and explicitly allowed. |
| Unmentioned | 937 | The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler. |
AI2Bot's three-year history
A site in the historical panel first named and disallowed this token in 2025-01; its most recent reading is 6.90%.
Allen Institute for AI's other tokens
A non-profit research institute that publishes its models and training corpora openly, which makes its crawlers unusual among those in this set.
| Token | Purpose | Published rate |
|---|---|---|
| AI2Bot-Dolma | Training | 4.62% |
Sites that treat these tokens differently
20 domains in the panel treat Allen Institute for AI's tokens differently from one another.
| Domain | AI2Bot | AI2Bot-Dolma |
|---|---|---|
| airbnb.com | FullBlock | Unmentioned |
| behance.net | FullBlock | Unmentioned |
| buzzfeed.com | Unmentioned | FullBlock |
| cookpad.com | FullBlock | Unmentioned |
| ea.com | FullBlock | Unmentioned |
| flashscore.com | FullBlock | Unmentioned |
| huffingtonpost.com | Unmentioned | FullBlock |
| huffpost.com | Unmentioned | FullBlock |
| indeed.com | PartialBlock | Unmentioned |
| jimdo.com | PartialBlock | Unmentioned |
| letterboxd.com | FullBlock | Unmentioned |
| onet.pl | Unmentioned | FullBlock |
| ouest-france.fr | Unmentioned | FullBlock |
| pimpbunny.com | FullBlock | Unmentioned |
| sharethrough.com | FullBlock | Unmentioned |
| spiegel.de | FullBlock | Unmentioned |
| tiktok.com | FullBlock | Unmentioned |
| tradingview.com | PartialBlock | Unmentioned |
| unesco.org | FullBlock | Unmentioned |
| uol.com.br | FullBlock | Unmentioned |