Purpose: Data broker. Crawls in order to sell the resulting datasets, which are widely used as AI training and retrieval data.
Crawls pages to build and sell structured data and a knowledge graph. It predates the current wave of model training and is a data business rather than a model developer, but is frequently blocked alongside AI crawlers.
9.65%
of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.
How sites treat this token
| Stance | Domains | What it means |
|---|---|---|
| FullBlock | 92 | Every path is disallowed for this token. |
| PartialBlock | 4 | Some paths are disallowed for this token; others are not. |
| Allowed | 4 | The token is named and explicitly allowed. |
| Unmentioned | 895 | The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler. |
Diffbot's three-year history
A site in the historical panel first named and disallowed this token in 2024-05; its most recent reading is 11.94%.