Bots on BareProxy.com

BareProxy AI Crawler Control Plugin: Allow, Block, Rate-Limit or Charge Each AI Bot

Thu, 08 Oct 2026 00:00:00 +0000

Status: planned, number 10 of 21 in BareProxy’s build order. The plugins are built easiest first, and this one is a day or two of coding: bot rules, address lists fetched and refreshed on a timer, counters. It comes after the Link previews plugin. This post describes what it will do, and it will be updated as it is built.

AI companies crawl the web to train models and to answer questions with fresh pages. GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and a growing list of others now make up a real share of the requests many sites serve. A site owner’s choices so far are thin. robots.txt is a polite request that some bots honor and some don’t, and it can’t say “yes, but slower” or “yes, if you pay”. Blocking by user agent is easy to get around, because anyone can send any user agent.

BareProxy AI Crawler Payment Gate Plugin: 402 Payment Required for Bots That Want the Content

Thu, 08 Oct 2026 00:00:00 +0000

Status: planned, number 19 of 21 in BareProxy’s build order. The plugins are built easiest first, and this one is about four or five days of coding: prices, proof of payment and books, on payment schemes still settling. It comes after the Response cache plugin. This post describes what it will do, and it will be updated as it is built.

AI companies need fresh, well-written pages, and a growing number of publishers want to be paid for them. The web has had a status code for this since 1997 that almost nobody used: 402 Payment Required. It is finally getting a job. A site answers a crawler with 402 and its terms; the crawler pays and comes back with proof; the site serves the page.

BareProxy Bot Challenges Plugin: A Proof-of-Work Check Before Scrapers Reach the Site

Thu, 08 Oct 2026 00:00:00 +0000

Status: planned, number 11 of 21 in BareProxy’s build order. The plugins are built easiest first, and this one is a day or two of coding: a challenge page, a check and a signed cookie. It comes after the AI crawler control plugin. This post describes what it will do, and it will be updated as it is built.

Not every bot announces itself. Scrapers that ignore robots.txt usually send a browser’s user agent and spread their requests over thousands of addresses, so neither user-agent rules nor rate limits catch them. Small sites, code forges and wikis have been hit hard by this, sometimes to the point of going offline.