Tallyhouse

Claude-Web

Anthropic — Uncertain

All crawlers · Agent Accessibility Index

Purpose not established

This token is tracked because sites still act on it, not because its purpose is known with confidence.

Purpose: Uncertain. Purpose not established. The token is named and blocked widely enough to track, but we cannot say with confidence what it is for — so the index counts it without claiming to know.

Named in many robots.txt files and blocked by 84 panel sites, but observed traffic appears minimal, so it may be superseded by the current tokens. Retained because sites clearly still act on it.

8.14%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock74Every path is disallowed for this token.
PartialBlock7Some paths are disallowed for this token; others are not.
Allowed9The token is named and explicitly allowed.
Unmentioned905The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

Claude-Web's three-year history

Claude-Web over time, 16 quarterly periods from 2023-01 to 2026-08.Claude-Web rose from 0.00% at 2023-01 to 9.02% at 2026-08.0%2%4%6%8%10%2023-012023-092024-052025-012025-102026-089.02%

A site in the historical panel first named and disallowed this token in 2023-09; its most recent reading is 9.02%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Anthropic's other tokens

Publishes separate tokens for training, search and user-initiated fetches, mirroring OpenAI's split. Two older tokens are still widely named in robots.txt files.

TokenPurposePublished rate
Claude-SearchBotSearch8.54%
Claude-UserUser-initiated8.04%
ClaudeBotTraining15.18%
anthropic-aiTraining9.35%

Sites that treat these tokens differently

164 domains in the panel treat Anthropic's tokens differently from one another. The first 25 are shown below.

DomainClaude-SearchBotClaude-UserClaude-WebClaudeBotanthropic-ai
abc.net.auUnmentionedUnmentionedFullBlockFullBlockFullBlock
academia.eduPartialBlockPartialBlockUnmentionedUnmentionedUnmentioned
ad.nlUnmentionedUnmentionedFullBlockFullBlockFullBlock
aftonbladet.seFullBlockUnmentionedFullBlockFullBlockFullBlock
airbnb.comUnmentionedUnmentionedUnmentionedFullBlockPartialBlock
aljazeera.comUnmentionedUnmentionedFullBlockFullBlockFullBlock
amap.comUnmentionedUnmentionedPartialBlockUnmentionedPartialBlock
amazon.caFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.jpFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.ukFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.co.zaFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.comFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.auFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.brFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.com.mxFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.deFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.esFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.frFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.inFullBlockFullBlockFullBlockFullBlockUnmentioned
amazon.itFullBlockFullBlockFullBlockFullBlockUnmentioned
amazonvideo.comUnmentionedFullBlockUnmentionedUnmentionedUnmentioned
amplitude.comUnmentionedUnmentionedUnmentionedPartialBlockPartialBlock
android.comPartialBlockUnmentionedUnmentionedPartialBlockPartialBlock
aol.comUnmentionedUnmentionedFullBlockFullBlockFullBlock
as.comFullBlockFullBlockUnmentionedFullBlockUnmentioned