Tallyhouse

Google-Extended

Google — Training consent

All crawlers · Agent Accessibility Index

Not a crawler

Google-Extended does not fetch pages. Disallowing it does not reduce crawling — it only withholds permission to train on content a conventional crawler already fetched.

Purpose: Training consent. Not a crawler at all. A signal withholding permission to train on content already fetched by a conventional crawler.

Not a crawler. A consent signal: disallowing it tells Google that content already fetched by Googlebot may not train Gemini. It does not reduce crawling and does not affect Search ranking, which is why sites unwilling to lose search traffic use it.

14.17%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock110Every path is disallowed for this token.
PartialBlock31Some paths are disallowed for this token; others are not.
Allowed33The token is named and explicitly allowed.
Unmentioned821The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

Google-Extended's three-year history

Google-Extended over time, 16 quarterly periods from 2023-01 to 2026-08.Google-Extended rose from 0.00% at 2023-01 to 16.18% at 2026-08.0%5%10%15%20%2023-012023-092024-052025-012025-102026-0816.18%

A site in the historical panel first named and disallowed this token in 2023-09; its most recent reading is 16.18%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Google's other tokens

The most fragmented operator in the set, with five tokens. Only Google-Extended is a consent signal; the others are crawlers serving distinct products.

TokenPurposePublished rate
Gemini-Deep-ResearchUser-initiated3.02%
Google-CloudVertexBotTraining5.13%
Google-NotebookLMUser-initiated2.61%
GoogleAgent-MarinerUser-initiated2.21%

Sites that treat these tokens differently

168 domains in the panel treat Google's tokens differently from one another. The first 25 are shown below.

DomainGemini-Deep-ResearchGoogle-CloudVertexBotGoogle-ExtendedGoogle-NotebookLMGoogleAgent-Mariner
abc.net.auUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
academia.eduPartialBlockUnmentionedFullBlockUnmentionedUnmentioned
ad.nlUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
aftonbladet.seUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
alibaba.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
amap.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
amazonvideo.comFullBlockFullBlockUnmentionedFullBlockFullBlock
amplitude.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
android.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
aol.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bandcamp.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bbc.co.ukUnmentionedFullBlockFullBlockUnmentionedUnmentioned
bbc.comUnmentionedFullBlockFullBlockUnmentionedUnmentioned
behance.netUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bfmtv.comFullBlockFullBlockFullBlockUnmentionedUnmentioned
bidmachine.ioUnmentionedUnmentionedAllowedUnmentionedUnmentioned
bitrix24.ruUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
bloomberg.comUnmentionedFullBlockFullBlockUnmentionedUnmentioned
branch.ioUnmentionedUnmentionedAllowedUnmentionedUnmentioned
buzzfeed.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
capcut.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
chatgpt.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
checkpoint.comUnmentionedUnmentionedAllowedUnmentionedUnmentioned
cisco.comUnmentionedUnmentionedAllowedUnmentionedUnmentioned
claude.aiUnmentionedUnmentionedFullBlockUnmentionedUnmentioned