Tallyhouse

cohere-training-data-crawler

Cohere — Training

All crawlers · Agent Accessibility Index

Purpose: Training. Bulk crawling to build model training data.

Cohere's crawler for collecting model training data, named explicitly for that purpose.

5.03%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock46Every path is disallowed for this token.
PartialBlock4Some paths are disallowed for this token; others are not.
Allowed2The token is named and explicitly allowed.
Unmentioned943The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

cohere-training-data-crawler's three-year history

cohere-training-data-crawler over time, 16 quarterly periods from 2023-01 to 2026-08.cohere-training-data-crawler rose from 0.00% at 2023-01 to 6.37% at 2026-08.0%2%4%6%8%2023-012023-092024-052025-012025-102026-086.37%

A site in the historical panel first named and disallowed this token in 2025-04; its most recent reading is 6.37%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Cohere's other tokens

An enterprise-focused model developer, now publishing an explicitly-named training crawler alongside an older token.

TokenPurposePublished rate
cohere-aiTraining9.75%

Sites that treat these tokens differently

65 domains in the panel treat Cohere's tokens differently from one another. The first 25 are shown below.

Domaincohere-aicohere-training-data-crawler
abc.net.auFullBlockUnmentioned
ad.nlFullBlockUnmentioned
airbnb.comPartialBlockUnmentioned
aljazeera.comFullBlockUnmentioned
amap.comPartialBlockUnmentioned
amplitude.comPartialBlockUnmentioned
android.comPartialBlockUnmentioned
aol.comFullBlockUnmentioned
apnews.comFullBlockUnmentioned
bbc.co.ukFullBlockUnmentioned
bbc.comFullBlockUnmentioned
behance.netUnmentionedFullBlock
bloomberg.comFullBlockUnmentioned
buzzfeed.comFullBlockUnmentioned
cloudflare.comAllowedUnmentioned
cnbc.comFullBlockUnmentioned
cookpad.comFullBlockUnmentioned
dagbladet.noFullBlockUnmentioned
dailymail.co.ukFullBlockUnmentioned
dailymail.comFullBlockUnmentioned
dcinside.comFullBlockUnmentioned
dw.comFullBlockUnmentioned
figma.comFullBlockUnmentioned
flashscore.comFullBlockUnmentioned
forbes.comFullBlockUnmentioned