Tallyhouse

meta-externalagent

Meta — Training

All crawlers · Agent Accessibility Index

Purpose: Training. Bulk crawling to build model training data.

Meta's crawler, collecting web content to train its models.

13.88%

of the panel names this token in robots.txt and disallows it, for the week of 2026-11-02.

How sites treat this token

StanceDomainsWhat it means
FullBlock118Every path is disallowed for this token.
PartialBlock19Some paths are disallowed for this token; others are not.
Allowed8The token is named and explicitly allowed.
Unmentioned842The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

meta-externalagent's three-year history

meta-externalagent over time, 16 quarterly periods from 2023-01 to 2026-08.meta-externalagent rose from 0.00% at 2023-01 to 14.06% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0814.06%

A site in the historical panel first named and disallowed this token in 2024-08; its most recent reading is 14.06%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Meta's other tokens

Publishes separate tokens for training, user-initiated fetches and indexing.

TokenPurposePublished rate
meta-externalfetcherUser-initiated6.99%
meta-webindexerSearch4.05%

Sites that treat these tokens differently

117 domains in the panel treat Meta's tokens differently from one another. The first 25 are shown below.

Domainmeta-externalagentmeta-externalfetchermeta-webindexer
abc.net.auFullBlockUnmentionedUnmentioned
ad.nlUnmentionedFullBlockUnmentioned
aftonbladet.seFullBlockUnmentionedUnmentioned
airbnb.comPartialBlockUnmentionedUnmentioned
amap.comPartialBlockUnmentionedUnmentioned
amplitude.comPartialBlockUnmentionedUnmentioned
ay267.comFullBlockUnmentionedUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockUnmentionedUnmentioned
bbc.comFullBlockUnmentionedUnmentioned
bblaa.comFullBlockUnmentionedUnmentioned
behance.netFullBlockUnmentionedUnmentioned
bloomberg.comFullBlockFullBlockUnmentioned
buzzfeed.comFullBlockFullBlockUnmentioned
canva.comFullBlockUnmentionedUnmentioned
capcut.comUnmentionedPartialBlockPartialBlock
cnbc.comFullBlockUnmentionedUnmentioned
cookpad.comFullBlockUnmentionedUnmentioned
corriere.itFullBlockUnmentionedUnmentioned
coursera.orgFullBlockUnmentionedUnmentioned
crn77.comFullBlockUnmentionedUnmentioned
dagbladet.noFullBlockFullBlockUnmentioned
dcinside.comFullBlockUnmentionedUnmentioned
disneyplus.comPartialBlockPartialBlockUnmentioned
doctolib.frPartialBlockUnmentionedUnmentioned