plugins

BareProxy AI Crawler Control Plugin: Allow, Block, Rate-Limit or Charge Each AI Bot

Status: built, number 10 of 21 in BareProxy’s build order, and out in BareProxy 0.12.0 as crawlers.wasm on the releases page. Its manual is the README in the code repository.

AI companies crawl the web to train models and to answer questions with fresh pages. GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and a growing list of others now make up a real share of the requests many sites serve. A site owner’s choices so far are thin. robots.txt is a polite request that some bots honor and some don’t, and it can’t say “yes, but slower”. Blocking by user agent is easy to get around, because anyone can send any user agent.

AI crawler control gives each bot its own answer, at the proxy, before the request reaches the site.

What It Does

  • Knows 24 AI bots by name, from OpenAI, Anthropic, Perplexity, Google, Common Crawl, Mistral AI, Amazon, Meta, ByteDance, DuckDuckGo, You.com and Ai2. Each is in a group: crawlers that collect pages for training, crawlers for AI search engines, and assistants that fetch a page because a person asked.
  • Allows, blocks or slows down each one, by name or by group, for the whole site or for one path. A rule can keep every training crawler out, let GPTBot in anyway, and hold PerplexityBot to 30 requests a minute.
  • Checks that a bot is who it says it is. Seven of the operators publish lists of the addresses their bots crawl from, and You.com states its one range. The plugin fetches those 13 lists, fetches them again every 6 hours, and checks each request’s address against them. A request that claims to be GPTBot from an address outside OpenAI’s list is a fake, and it’s turned away.
  • Keeps a log of who is scraping what. Every bot request is noted in its record and counted by bot and answer. A report page shows each bot’s requests today and yesterday, and the parts of the site it read.

Here is a config:

plugin crawlers /etc/bareproxy/plugins/crawlers.wasm
  config /etc/bareproxy/plugins/crawlers.conf
  allow-http openai.com:443 claude.com:443 www.perplexity.ai:443 developers.google.com:443 index.commoncrawl.org:443 mistral.ai:443 developer.amazon.com:443
  store 1MB
  on-error open

site example.com
  use crawlers
  route /* -> files /var/www/example/public

And the plugin’s own:

# crawlers.conf
block training
allow GPTBot
limit PerplexityBot 30 per 1m
report /.crawlers

allow-http names the hosts the lists come from, and the plugin can call nothing else. store keeps the lists across restarts. on-error open matters here: if the plugin ever fails, the site keeps serving people, and only the bot rules pause.

Fakes Are Turned Away

Scrapers often borrow a famous bot’s name, hoping a site lets that bot in. With the published lists, the plugin can tell them apart. One instance of the plugin fetches each list for all the others, and with a store the lists survive a restart, so the check works from the first request after one.

Some operators publish no list. Meta names its network but gives nothing to fetch, and ByteDance gives nothing either. For those bots the plugin can’t check, and the record says so. A site can give such a bot addresses of its own, or treat every claim it can’t check as a fake.

The same check catches fake Googlebots, an old trick for getting past limits. Two lines in the plugin’s config add Googlebot with Google’s own address list.

What why Says

Every decision lands in the request’s record, so why can tell the story of any bot request:

GPTBot (OpenAI, training): 20.171.206.5 is in OpenAI's list; let in by line 2, its 41st request let in this hour
fake GPTBot: 203.0.113.9 isn't in OpenAI's list (18 ranges); turned away (fake block)

A bot that gets in carries two request headers, X-Bot with its name and X-Bot-Check with the result of the check, so the app and BareProxy’s own route rules can tell a verified bot from anyone else. A client that sends those headers itself has them dropped.

Two More Plugins Build on It

Crawler control is the first of three plugins that decide what bots see. The AI crawler payment gate will quote prices to bots, and it leans on crawler control to tell real crawlers from impostors before it does. RenderCache decides what an admitted bot sees: a fully rendered page instead of an empty JavaScript shell. Both come later in the build order.

The whole program is on the plugins page.