Purpose: Archive. Crawls to build a public web archive; the corpus is separately used for training.
Common Crawl's crawler, running since 2011 for a non-profit web archive. Nothing about it changed when AI training became contentious; what changed is that sites began reading it as an AI crawler, because its archive is a common training corpus. It is now among the most blocked agents in the panel despite being the oldest and the only non-commercial one.
16.38%
of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.
How sites treat this token
| Stance | Domains | What it means |
|---|---|---|
| FullBlock | 145 | Every path is disallowed for this token. |
| PartialBlock | 18 | Some paths are disallowed for this token; others are not. |
| Allowed | 19 | The token is named and explicitly allowed. |
| Unmentioned | 813 | The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler. |
CCBot's three-year history
A site in the historical panel first named and disallowed this token in 2023-03; its most recent reading is 17.77%.