Tallyhouse

OAI-SearchBot

OpenAI — Search

All crawlers · Agent Accessibility Index

Purpose: Search. Indexes pages so an answer engine can retrieve and cite them.

Indexes pages so they can appear as results and citations in ChatGPT search. It has the most even split in the panel between full and partial blocks, suggesting sites weigh it more like a search engine than a training crawler.

10.05%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock51Every path is disallowed for this token.
PartialBlock49Some paths are disallowed for this token; others are not.
Allowed31The token is named and explicitly allowed.
Unmentioned864The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

OAI-SearchBot's three-year history

OAI-SearchBot over time, 16 quarterly periods from 2023-01 to 2026-08.OAI-SearchBot rose from 0.00% at 2023-01 to 11.94% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0811.94%

A site in the historical panel first named and disallowed this token in 2024-08; its most recent reading is 11.94%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

OpenAI's other tokens

Operates three separately-named tokens for three different jobs, which is why sites so often treat them differently.

TokenPurposePublished rate
ChatGPT-UserUser-initiated11.16%
GPTBotTraining16.58%

Sites that treat these tokens differently

125 domains in the panel treat OpenAI's tokens differently from one another. The first 25 are shown below.

DomainChatGPT-UserGPTBotOAI-SearchBot
abc.net.auFullBlockFullBlockUnmentioned
academia.eduPartialBlockFullBlockPartialBlock
aftonbladet.seUnmentionedUnmentionedPartialBlock
airbnb.comPartialBlockFullBlockPartialBlock
alibaba.comUnmentionedFullBlockUnmentioned
aljazeera.comFullBlockFullBlockUnmentioned
amazonvideo.comFullBlockUnmentionedUnmentioned
android.comUnmentionedPartialBlockPartialBlock
aol.comPartialBlockFullBlockUnmentioned
apnews.comUnmentionedFullBlockUnmentioned
archiveofourown.orgFullBlockFullBlockUnmentioned
bandcamp.comUnmentionedFullBlockUnmentioned
behance.netUnmentionedFullBlockUnmentioned
bidmachine.ioUnmentionedFullBlockAllowed
bloomberg.comAllowedFullBlockAllowed
branch.ioUnmentionedAllowedUnmentioned
bt.dkUnmentionedFullBlockUnmentioned
buzzfeed.comUnmentionedFullBlockUnmentioned
cambridge.orgFullBlockUnmentionedUnmentioned
canva.comUnmentionedFullBlockUnmentioned
cbsnews.comUnmentionedFullBlockUnmentioned
cisco.comUnmentionedAllowedUnmentioned
cloudflare.comAllowedAllowedUnmentioned
cookpad.comFullBlockFullBlockUnmentioned
corriere.itFullBlockFullBlockAllowed