Tallyhouse

omgili

Webz.io — Data broker

All crawlers · Agent Accessibility Index

Purpose: Data broker. Crawls in order to sell the resulting datasets, which are widely used as AI training and retrieval data.

An older Webz.io token. Webz.io crawls the web and sells the resulting datasets, which are widely used as AI training and retrieval data. The most-blocked untracked token found in the audit, at 96 panel sites.

9.45%

of the panel names this token in robots.txt and disallows it, for the week of 2026-10-05.

How sites treat this token

StanceDomainsWhat it means
FullBlock92Every path is disallowed for this token.
PartialBlock2Some paths are disallowed for this token; others are not.
Allowed3The token is named and explicitly allowed.
Unmentioned898The file never names this token. This is not consent — the site may still close it off with a blanket rule for every crawler.

omgili's three-year history

omgili over time, 16 quarterly periods from 2023-01 to 2026-08.omgili rose from 0.26% at 2023-01 to 10.88% at 2026-08.0%5%10%15%2023-012023-092024-052025-012025-102026-0810.88%

A site in the historical panel first named and disallowed this token in 2023-01; its most recent reading is 10.88%.

Note: this chart reads Common Crawl's own 611-domain historical panel, not the panel behind the stance table above — a different population from the live index panel, so levels are not comparable between the two indices.

Webz.io's other tokens

A data broker selling web datasets, widely used as AI training and retrieval data.

TokenPurposePublished rate
omgilibotData broker7.94%
webzio-extendedData broker5.73%

Sites that treat these tokens differently

72 domains in the panel treat Webz.io's tokens differently from one another. The first 25 are shown below.

Domainomgiliomgilibotwebzio-extended
airbnb.comUnmentionedUnmentionedFullBlock
amazon.caFullBlockUnmentionedFullBlock
amazon.co.jpFullBlockUnmentionedFullBlock
amazon.co.ukFullBlockUnmentionedFullBlock
amazon.co.zaFullBlockUnmentionedFullBlock
amazon.comFullBlockUnmentionedFullBlock
amazon.com.auFullBlockUnmentionedFullBlock
amazon.com.brFullBlockUnmentionedFullBlock
amazon.com.mxFullBlockUnmentionedFullBlock
amazon.deFullBlockUnmentionedFullBlock
amazon.esFullBlockUnmentionedFullBlock
amazon.frFullBlockUnmentionedFullBlock
amazon.inFullBlockUnmentionedFullBlock
amazon.itFullBlockUnmentionedFullBlock
amazonvideo.comFullBlockUnmentionedFullBlock
aol.comFullBlockFullBlockUnmentioned
bandcamp.comFullBlockUnmentionedUnmentioned
bbc.co.ukFullBlockFullBlockUnmentioned
bbc.comFullBlockFullBlockUnmentioned
bol.comFullBlockUnmentionedUnmentioned
buzzfeed.comUnmentionedFullBlockUnmentioned
canva.comFullBlockUnmentionedFullBlock
chatgpt.comFullBlockFullBlockUnmentioned
cnbc.comFullBlockFullBlockUnmentioned
cookpad.comFullBlockFullBlockUnmentioned