Tallyhouse

omgilibot

Webz.io — Data broker

All crawlers · Agent Accessibility Index

Purpose: Data broker. Crawls in order to sell the resulting datasets, which are widely used as AI training and retrieval data.

A Webz.io token, sibling to omgili; both are still widely named in robots.txt.

7.94%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock76Every path is disallowed for this token.
PartialBlock3Some paths are disallowed for this token; others are not.
Allowed2The token is named and explicitly allowed.
Unmentioned914The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

omgilibot's three-year history

omgilibot over time, 16 quarterly periods from 2023-01 to 2026-08.omgilibot rose from 0.26% at 2023-01 to 7.43% at 2026-08.0%2%4%6%8%10%2023-012023-092024-052025-012025-102026-087.43%

A site in the historical panel first named and disallowed this token in 2023-01; its most recent reading is 7.43%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Webz.io's other tokens

A data broker selling web datasets, widely used as AI training and retrieval data.

TokenPurposePublished rate
omgiliData broker9.45%
webzio-extendedData broker5.73%

Sites that treat these tokens differently

72 domains in the panel treat Webz.io's tokens differently from one another. The first 25 are shown below.

Domainomgiliomgilibotwebzio-extended
airbnb.comUnmentionedUnmentionedFullBlock
amazon.caFullBlockUnmentionedFullBlock
amazon.co.jpFullBlockUnmentionedFullBlock
amazon.co.ukFullBlockUnmentionedFullBlock
amazon.co.zaFullBlockUnmentionedFullBlock
amazon.comFullBlockUnmentionedFullBlock
amazon.com.auFullBlockUnmentionedFullBlock
amazon.com.brFullBlockUnmentionedFullBlock
amazon.com.mxFullBlockUnmentionedFullBlock
amazon.deFullBlockUnmentionedFullBlock
amazon.esFullBlockUnmentionedFullBlock
amazon.frFullBlockUnmentionedFullBlock
amazon.inFullBlockUnmentionedFullBlock
amazon.itFullBlockUnmentionedFullBlock
amazonvideo.comFullBlockUnmentionedFullBlock
aol.comFullBlockFullBlockUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockFullBlockUnmentioned
bbc.comFullBlockFullBlockUnmentioned
bol.comFullBlockUnmentionedUnmentioned
buzzfeed.comUnmentionedFullBlockUnmentioned
canva.comFullBlockUnmentionedFullBlock
chatgpt.comFullBlockFullBlockUnmentioned
cnbc.comFullBlockFullBlockUnmentioned
cookpad.comFullBlockFullBlockUnmentioned