Tallyhouse

Google-NotebookLM

Google — User-initiated

All crawlers · Agent Accessibility Index

Purpose: User-initiated. Fetches a specific page because a user asked, rather than crawling broadly. Blocking this stops people using an assistant to read the site; it does not prevent training.

Fetches a page a NotebookLM user has pointed it at, rather than crawling broadly.

2.61%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock21Every path is disallowed for this token.
PartialBlock5Some paths are disallowed for this token; others are not.
Allowed0The token is named and explicitly allowed.
Unmentioned969The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

Google-NotebookLM's three-year history

Google-NotebookLM over time, 16 quarterly periods from 2023-01 to 2026-08.Google-NotebookLM rose from 0.00% at 2023-01 to 3.71% at 2026-08.0%1%2%3%4%2023-012023-092024-052025-012025-102026-083.71%

A site in the historical panel first named and disallowed this token in 2025-10; its most recent reading is 3.71%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Google's other tokens

The most fragmented operator in the set, with five tokens. Only Google-Extended is a consent signal; the others are crawlers serving distinct products.

TokenPurposePublished rate
Gemini-Deep-ResearchUser-initiated3.02%
Google-CloudVertexBotTraining5.13%
Google-ExtendedTraining consent14.17%
GoogleAgent-MarinerUser-initiated2.21%

Sites that treat these tokens differently

168 domains in the panel treat Google's tokens differently from one another. The first 25 are shown below.

DomainGemini-Deep-ResearchGoogle-CloudVertexBotGoogle-ExtendedGoogle-NotebookLMGoogleAgent-Mariner
abc.net.auUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
academia.eduPartialBlockUnmentionedFullBlockUnmentionedUnmentioned
ad.nlUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
aftonbladet.seUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
alibaba.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
amap.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
amazonvideo.comFullBlockFullBlockUnmentionedFullBlockFullBlock
amplitude.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
android.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
aol.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bandcamp.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bbc.co.ukUnmentionedFullBlockFullBlockUnmentionedUnmentioned
bbc.comUnmentionedFullBlockFullBlockUnmentionedUnmentioned
behance.netUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
bfmtv.comFullBlockFullBlockFullBlockUnmentionedUnmentioned
bidmachine.ioUnmentionedUnmentionedAllowedUnmentionedUnmentioned
bitrix24.ruUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
bloomberg.comUnmentionedFullBlockFullBlockUnmentionedUnmentioned
branch.ioUnmentionedUnmentionedAllowedUnmentionedUnmentioned
buzzfeed.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
capcut.comUnmentionedUnmentionedPartialBlockUnmentionedUnmentioned
chatgpt.comUnmentionedUnmentionedFullBlockUnmentionedUnmentioned
checkpoint.comUnmentionedUnmentionedAllowedUnmentionedUnmentioned
cisco.comUnmentionedUnmentionedAllowedUnmentionedUnmentioned
claude.aiUnmentionedUnmentionedFullBlockUnmentionedUnmentioned