Tallyhouse

meta-externalfetcher

Meta — User-initiated

All crawlers · Agent Accessibility Index

Purpose: User-initiated. Fetches a specific page because a user asked, rather than crawling broadly. Blocking this stops people using an assistant to read the site; it does not prevent training.

Fetches a page because a Meta AI user asked for it, rather than as part of a bulk crawl.

6.53%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock55Every path is disallowed for this token.
PartialBlock10Some paths are disallowed for this token; others are not.
Allowed2The token is named and explicitly allowed.
Unmentioned928The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

meta-externalfetcher's three-year history

meta-externalfetcher over time, 16 quarterly periods from 2023-01 to 2026-08.meta-externalfetcher rose from 0.00% at 2023-01 to 7.16% at 2026-08.0%2%4%6%8%2023-012023-092024-052025-012025-102026-087.16%

A site in the historical panel first named and disallowed this token in 2024-11; its most recent reading is 7.16%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Meta's other tokens

Publishes separate tokens for training, user-initiated fetches and indexing.

TokenPurposePublished rate
meta-externalagentTraining11.76%
meta-webindexerSearch3.32%

Sites that treat these tokens differently

105 domains in the panel treat Meta's tokens differently from one another. The first 25 are shown below.

Domainmeta-externalagentmeta-externalfetchermeta-webindexer
abc.net.auFullBlockUnmentionedUnmentioned
ad.nlUnmentionedFullBlockUnmentioned
aftonbladet.seFullBlockUnmentionedUnmentioned
airbnb.comPartialBlockUnmentionedUnmentioned
amap.comPartialBlockUnmentionedUnmentioned
amplitude.comPartialBlockUnmentionedUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockUnmentionedUnmentioned
bbc.comFullBlockUnmentionedUnmentioned
behance.netFullBlockUnmentionedUnmentioned
bloomberg.comFullBlockFullBlockUnmentioned
buzzfeed.comFullBlockFullBlockUnmentioned
canva.comFullBlockUnmentionedUnmentioned
capcut.comUnmentionedPartialBlockPartialBlock
cnbc.comFullBlockUnmentionedUnmentioned
cookpad.comFullBlockUnmentionedUnmentioned
corriere.itFullBlockUnmentionedUnmentioned
coursera.orgFullBlockUnmentionedUnmentioned
dagbladet.noFullBlockUnmentionedUnmentioned
dailymail.co.ukFullBlockFullBlockUnmentioned
dailymail.comFullBlockFullBlockUnmentioned
dcinside.comFullBlockUnmentionedUnmentioned
disneyplus.comPartialBlockPartialBlockUnmentioned
doctolib.frPartialBlockUnmentionedUnmentioned
dw.comFullBlockFullBlockUnmentioned