Tallyhouse

ChatGPT-User

OpenAI — User-initiated

All crawlers · Agent Accessibility Index

Purpose: User-initiated. Fetches a specific page because a user asked, rather than crawling broadly. Blocking this stops people using an assistant to read the site; it does not prevent training.

Fetches a page because a ChatGPT user asked for it. Not a bulk crawler and not collected for training. Sites wanting to remain usable by people using AI assistants, without being training data, allow this and block GPTBot.

11.16%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock68Every path is disallowed for this token.
PartialBlock43Some paths are disallowed for this token; others are not.
Allowed29The token is named and explicitly allowed.
Unmentioned855The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

ChatGPT-User's three-year history

ChatGPT-User over time, 16 quarterly periods from 2023-01 to 2026-08.ChatGPT-User rose from 0.00% at 2023-01 to 13.00% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0813.00%

A site in the historical panel first named and disallowed this token in 2023-03; its most recent reading is 13.00%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

OpenAI's other tokens

Operates three separately-named tokens for three different jobs, which is why sites so often treat them differently.

TokenPurposePublished rate
GPTBotTraining16.58%
OAI-SearchBotSearch10.05%

Sites that treat these tokens differently

125 domains in the panel treat OpenAI's tokens differently from one another. The first 25 are shown below.

DomainChatGPT-UserGPTBotOAI-SearchBot
abc.net.auFullBlockFullBlockUnmentioned
academia.eduPartialBlockFullBlockPartialBlock
aftonbladet.seUnmentionedUnmentionedPartialBlock
airbnb.comPartialBlockFullBlockPartialBlock
alibaba.comUnmentionedFullBlockUnmentioned
aljazeera.comFullBlockFullBlockUnmentioned
amazonvideo.comFullBlockUnmentionedUnmentioned
android.comUnmentionedPartialBlockPartialBlock
aol.comPartialBlockFullBlockUnmentioned
apnews.comUnmentionedFullBlockUnmentioned
archiveofourown.orgFullBlockFullBlockUnmentioned
bandcamp.comUnmentionedFullBlockUnmentioned
behance.netUnmentionedFullBlockUnmentioned
bidmachine.ioUnmentionedFullBlockAllowed
bloomberg.comAllowedFullBlockAllowed
branch.ioUnmentionedAllowedUnmentioned
bt.dkUnmentionedFullBlockUnmentioned
buzzfeed.comUnmentionedFullBlockUnmentioned
cambridge.orgFullBlockUnmentionedUnmentioned
canva.comUnmentionedFullBlockUnmentioned
cbsnews.comUnmentionedFullBlockUnmentioned
cisco.comUnmentionedAllowedUnmentioned
cloudflare.comAllowedAllowedUnmentioned
cookpad.comFullBlockFullBlockUnmentioned
corriere.itFullBlockFullBlockAllowed