AI Crawlers

Allen Institute for AI

A non-profit research institute that publishes its models and training corpora openly, which makes its crawlers unusual among those in this set.

20 domains in the panel set Allen Institute for AI's tokens differently from one another.

Amazon

Crawls for product, search and assistant features.

Anthropic

Publishes separate tokens for training, search and user-initiated fetches, mirroring OpenAI's split. Two older tokens are still widely named in robots.txt files.

164 domains in the panel set Anthropic's tokens differently from one another.

Apple

Applebot-Extended is a consent signal, not a crawler: it governs training use of content Applebot has already fetched.

Bright Data

A data broker selling web datasets.

ByteDance

Operator of TikTok. Bytespider has been widely reported as crawling aggressively.

Cognition

Builds Devin, an AI software engineering agent.

Cohere

An enterprise-focused model developer, now publishing an explicitly-named training crawler alongside an older token.

65 domains in the panel set Cohere's tokens differently from one another.

Common Crawl

A non-profit web archive operating since 2011, long predating the current AI industry. Its corpus is widely used as training data, which is why blocking it became common only recently.

DeepSeek

A Chinese model developer.

Diffbot

A commercial web-extraction service selling structured data and a knowledge graph built from crawled pages.

DuckDuckGo

A privacy-focused search engine, now also serving AI-assisted answers.

Google

The most fragmented operator in the set, with five tokens. Only Google-Extended is a consent signal; the others are crawlers serving distinct products.

168 domains in the panel set Google's tokens differently from one another.

Hive

Operates ImageSift, an image-focused crawler used to build image datasets.

Huawei

Publisher of the PanGu model family.

Meta

Publishes separate tokens for training, user-initiated fetches and indexing.

105 domains in the panel set Meta's tokens differently from one another.

Mistral

A European model developer.

OpenAI

Operates three separately-named tokens for three different jobs, which is why sites so often treat them differently.

125 domains in the panel set OpenAI's tokens differently from one another.

Perplexity

An answer engine, operating a crawler and a user-initiated fetcher.

88 domains in the panel set Perplexity's tokens differently from one another.

Phind

An answer engine aimed at developers.

SB Intuitions

A SoftBank subsidiary building Japanese-language models.

Tavily

Sells a search API that supplies web results to other companies' AI agents.

Timpi

Runs a decentralised crawling network gathering training data.

Webz.io

A data broker selling web datasets, widely used as AI training and retrieval data.

72 domains in the panel set Webz.io's tokens differently from one another.

Wrtn

A Korean AI assistant.

You.com

An answer engine.

Zhipu AI

A Chinese model developer, publisher of the ChatGLM models.

iAsk

An answer engine.