Tallyhouse

meta-externalagent

Meta — Training

All crawlers · Agent Accessibility Index

Purpose: Training. Bulk crawling to build model training data.

Meta's crawler, collecting web content to train its models.

11.76%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock99Every path is disallowed for this token.
PartialBlock18Some paths are disallowed for this token; others are not.
Allowed7The token is named and explicitly allowed.
Unmentioned871The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

meta-externalagent's three-year history

meta-externalagent over time, 16 quarterly periods from 2023-01 to 2026-08.meta-externalagent rose from 0.00% at 2023-01 to 14.06% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0814.06%

A site in the historical panel first named and disallowed this token in 2024-08; its most recent reading is 14.06%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Meta's other tokens

Publishes separate tokens for training, user-initiated fetches and indexing.

TokenPurposePublished rate
meta-externalfetcherUser-initiated6.53%
meta-webindexerSearch3.32%

Sites that treat these tokens differently

105 domains in the panel treat Meta's tokens differently from one another. The first 25 are shown below.

Domainmeta-externalagentmeta-externalfetchermeta-webindexer
abc.net.auFullBlockUnmentionedUnmentioned
ad.nlUnmentionedFullBlockUnmentioned
aftonbladet.seFullBlockUnmentionedUnmentioned
airbnb.comPartialBlockUnmentionedUnmentioned
amap.comPartialBlockUnmentionedUnmentioned
amplitude.comPartialBlockUnmentionedUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockUnmentionedUnmentioned
bbc.comFullBlockUnmentionedUnmentioned
behance.netFullBlockUnmentionedUnmentioned
bloomberg.comFullBlockFullBlockUnmentioned
buzzfeed.comFullBlockFullBlockUnmentioned
canva.comFullBlockUnmentionedUnmentioned
capcut.comUnmentionedPartialBlockPartialBlock
cnbc.comFullBlockUnmentionedUnmentioned
cookpad.comFullBlockUnmentionedUnmentioned
corriere.itFullBlockUnmentionedUnmentioned
coursera.orgFullBlockUnmentionedUnmentioned
dagbladet.noFullBlockUnmentionedUnmentioned
dailymail.co.ukFullBlockFullBlockUnmentioned
dailymail.comFullBlockFullBlockUnmentioned
dcinside.comFullBlockUnmentionedUnmentioned
disneyplus.comPartialBlockPartialBlockUnmentioned
doctolib.frPartialBlockUnmentionedUnmentioned
dw.comFullBlockFullBlockUnmentioned