Tallyhouse

meta-webindexer

Meta — Search

All crawlers · Agent Accessibility Index

Purpose: Search. Indexes pages so an answer engine can retrieve and cite them.

Indexes pages for retrieval within Meta's products.

3.32%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock29Every path is disallowed for this token.
PartialBlock4Some paths are disallowed for this token; others are not.
Allowed1The token is named and explicitly allowed.
Unmentioned961The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

meta-webindexer's three-year history

meta-webindexer over time, 16 quarterly periods from 2023-01 to 2026-08.meta-webindexer rose from 0.00% at 2023-01 to 3.98% at 2026-08.0%1%2%3%4%2023-012023-092024-052025-012025-102026-083.98%

A site in the historical panel first named and disallowed this token in 2025-10; its most recent reading is 3.98%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Meta's other tokens

Publishes separate tokens for training, user-initiated fetches and indexing.

TokenPurposePublished rate
meta-externalagentTraining11.76%
meta-externalfetcherUser-initiated6.53%

Sites that treat these tokens differently

105 domains in the panel treat Meta's tokens differently from one another. The first 25 are shown below.

Domainmeta-externalagentmeta-externalfetchermeta-webindexer
abc.net.auFullBlockUnmentionedUnmentioned
ad.nlUnmentionedFullBlockUnmentioned
aftonbladet.seFullBlockUnmentionedUnmentioned
airbnb.comPartialBlockUnmentionedUnmentioned
amap.comPartialBlockUnmentionedUnmentioned
amplitude.comPartialBlockUnmentionedUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockUnmentionedUnmentioned
bbc.comFullBlockUnmentionedUnmentioned
behance.netFullBlockUnmentionedUnmentioned
bloomberg.comFullBlockFullBlockUnmentioned
buzzfeed.comFullBlockFullBlockUnmentioned
canva.comFullBlockUnmentionedUnmentioned
capcut.comUnmentionedPartialBlockPartialBlock
cnbc.comFullBlockUnmentionedUnmentioned
cookpad.comFullBlockUnmentionedUnmentioned
corriere.itFullBlockUnmentionedUnmentioned
coursera.orgFullBlockUnmentionedUnmentioned
dagbladet.noFullBlockUnmentionedUnmentioned
dailymail.co.ukFullBlockFullBlockUnmentioned
dailymail.comFullBlockFullBlockUnmentioned
dcinside.comFullBlockUnmentionedUnmentioned
disneyplus.comPartialBlockPartialBlockUnmentioned
doctolib.frPartialBlockUnmentionedUnmentioned
dw.comFullBlockFullBlockUnmentioned