Tallyhouse

Claude-User

Anthropic — User-initiated

All crawlers · Agent Accessibility Index

Purpose: User-initiated. Fetches a specific page because a user asked, rather than crawling broadly. Blocking this stops people using an assistant to read the site; it does not prevent training.

Fetches a page because a Claude user asked for it, rather than as part of a bulk crawl.

9.02%

of the panel names this token in robots.txt and disallows it, for the week of 2026-11-02.

How sites treat this token

StanceDomainsWhat it means
FullBlock63Every path is disallowed for this token.
PartialBlock26Some paths are disallowed for this token; others are not.
Allowed13The token is named and explicitly allowed.
Unmentioned885The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

Claude-User's three-year history

Claude-User over time, 16 quarterly periods from 2023-01 to 2026-08.Claude-User rose from 0.00% at 2023-01 to 10.88% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0810.88%

A site in the historical panel first named and disallowed this token in 2025-04; its most recent reading is 10.88%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Anthropic's other tokens

Publishes separate tokens for training, search and user-initiated fetches, mirroring OpenAI's split. Two older tokens are still widely named in robots.txt files.

TokenPurposePublished rate
Claude-SearchBotSearch9.63%
Claude-WebUncertain10.13%
ClaudeBotTraining17.22%
anthropic-aiTraining10.94%

Sites that treat these tokens differently

176 domains in the panel treat Anthropic's tokens differently from one another. The first 25 are shown below.

DomainClaude-SearchBotClaude-UserClaude-WebClaudeBotanthropic-ai
abc.net.auUnmentionedUnmentionedFullBlockFullBlockFullBlock
academia.eduPartialBlockPartialBlockUnmentionedUnmentionedUnmentioned
ad.nlUnmentionedUnmentionedFullBlockFullBlockFullBlock
aftonbladet.seFullBlockUnmentionedFullBlockFullBlockFullBlock
airbnb.comUnmentionedUnmentionedUnmentionedFullBlockPartialBlock
aliexpress.comUnmentionedUnmentionedUnmentionedFullBlockUnmentioned
aljazeera.comUnmentionedUnmentionedFullBlockFullBlockFullBlock
amap.comUnmentionedUnmentionedPartialBlockUnmentionedPartialBlock
amazon.caFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.jpFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.ukFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.zaFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.comFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.auFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.brFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.mxFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.deFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.esFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.frFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.inFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.itFullBlockFullBlockFullBlockFullBlockUnmentioned
amazonvideo.comUnmentionedFullBlockUnmentionedUnmentionedUnmentioned
amplitude.comUnmentionedUnmentionedUnmentionedPartialBlockPartialBlock
android.comPartialBlockUnmentionedUnmentionedPartialBlockPartialBlock
aol.comUnmentionedUnmentionedFullBlockFullBlockFullBlock