Purpose: Data broker. Crawls in order to sell the resulting datasets, which are widely used as AI training and retrieval data.
Bright Data crawls the web and sells the resulting datasets. Like Diffbot and Webz.io it is a data business rather than a model developer, and is blocked alongside AI crawlers for the same reason.
3.42%
of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.
How sites treat this token
| Stance | Domains | What it means |
|---|---|---|
| FullBlock | 33 | Every path is disallowed for this token. |
| PartialBlock | 1 | Some paths are disallowed for this token; others are not. |
| Allowed | 0 | The token is named and explicitly allowed. |
| Unmentioned | 961 | The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler. |
Brightbot's three-year history
A site in the historical panel first named and disallowed this token in 2025-07; its most recent reading is 4.77%.