Three years ago almost no major website told AI crawlers to stay out. By 2026-08, more than a quarter did — and the change was not gradual.
A fixed panel of 611 long-running popular sites, read from Common Crawl's archive, stood at 1.28% in 2023-01. The largest single move in the series is 2023-09, up 13.65 points on the reading before it — the first snapshot taken after OpenAI launched GPTBot in August 2023. The figure reached 27.85% by 2026-08.
Trend
- Targeted — named crawler, disallowed
- Effective — named or blanket disallow
The weekly Agent Accessibility Index currently reads 24.32% for the week of 2026-10-05.
Read Three years of AI-crawler blocking for the full three-year series and the method behind it.