Tallyhouse

AI2Bot-Dolma

Allen Institute for AI — Training

All crawlers · Agent Accessibility Index

Purpose: Training. Bulk crawling to build model training data.

Collects content for Dolma, the Allen Institute's openly published training corpus.

4.62%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock45Every path is disallowed for this token.
PartialBlock1Some paths are disallowed for this token; others are not.
Allowed2The token is named and explicitly allowed.
Unmentioned947The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

AI2Bot-Dolma's three-year history

AI2Bot-Dolma over time, 16 quarterly periods from 2023-01 to 2026-08.AI2Bot-Dolma rose from 0.00% at 2023-01 to 4.51% at 2026-08.0%1%2%3%4%5%2023-012023-092024-052025-012025-102026-084.51%

A site in the historical panel first named and disallowed this token in 2025-01; its most recent reading is 4.51%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Allen Institute for AI's other tokens

A non-profit research institute that publishes its models and training corpora openly, which makes its crawlers unusual among those in this set.

TokenPurposePublished rate
AI2BotTraining5.63%

Sites that treat these tokens differently

20 domains in the panel treat Allen Institute for AI's tokens differently from one another.

DomainAI2BotAI2Bot-Dolma
airbnb.comFullBlockUnmentioned
behance.netFullBlockUnmentioned
buzzfeed.comUnmentionedFullBlock
cookpad.comFullBlockUnmentioned
ea.comFullBlockUnmentioned
flashscore.comFullBlockUnmentioned
huffingtonpost.comUnmentionedFullBlock
huffpost.comUnmentionedFullBlock
indeed.comPartialBlockUnmentioned
jimdo.comPartialBlockUnmentioned
letterboxd.comFullBlockUnmentioned
onet.plUnmentionedFullBlock
ouest-france.frUnmentionedFullBlock
pimpbunny.comFullBlockUnmentioned
sharethrough.comFullBlockUnmentioned
spiegel.deFullBlockUnmentioned
tiktok.comFullBlockUnmentioned
tradingview.comPartialBlockUnmentioned
unesco.orgFullBlockUnmentioned
uol.com.brFullBlockUnmentioned