BareProxy Adds AI Crawler Control That Tells Real GPTBot and ClaudeBot Requests From Fakes
BareProxy 0.12.0 is out today with its tenth plugin, AI crawler control. It decides which AI bots get into a site and how often. Before it decides, it checks that each bot is who it says it is. A block list of user agents can’t do that. Any scraper can put GPTBot or ClaudeBot in its User-Agent, and some do, hoping for the welcome those bots get.
The plugin is crawlers.wasm on the releases page, added to a site with a few lines of BareProxy config. Its page on this site has the details, and its manual is in the code repository.
Who Gets In
Each of the 24 bots the plugin knows belongs to a group. Training crawlers collect pages to train AI models. Search crawlers index pages for AI answer engines, which link back to them. The third group, assistants, fetch one page at a time because a person asked ChatGPT or Claude about it. A rule can name a bot or a group, for the whole site or for one path:
block training
allow GPTBot
limit PerplexityBot 30 per 1m
Those three lines keep every training crawler out except GPTBot, and hold PerplexityBot to 30 requests a minute. Search crawlers and assistants get in, since no rule names them. /robots.txt stays open to every bot, so the ones turned away can still read why.
The Bots It Knows
| Company | Bots | Checked against |
|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI’s published lists |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Anthropic’s published list |
| Perplexity | PerplexityBot, Perplexity-User | Perplexity’s published lists |
| Google-Agent, Google-GeminiNotebook | Google’s list, for Google-Agent | |
| Common Crawl | CCBot | Common Crawl’s published list |
| Mistral AI | MistralAI-Index, MistralAI-User, MistralAI-Training | Mistral’s lists, for Index and User |
| Amazon | Amazonbot, Amzn-SearchBot, Amzn-User | Amazon’s published lists |
| Meta | Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher | No list published |
| ByteDance | Bytespider | No list published |
| DuckDuckGo | DuckAssistBot | No list to fetch |
| You.com | YouBot | The one range You.com states |
| Ai2 | AI2Bot | No list published |
Telling Real Bots From Fakes
Seven of these companies publish the addresses their crawlers use, in 13 lists. The plugin fetches them a second after it starts and again every six hours. One copy of the plugin does the fetching for all the others (BareProxy runs several copies to share the load), and with a store the lists survive a restart, so the check works from the very first request after one.
A request that says it’s GPTBot from an address outside OpenAI’s list isn’t GPTBot. The plugin calls it a fake and, by default, answers 403.
Some companies give nothing to check against. Meta names its network but publishes no list, and ByteDance gives nothing for Bytespider. Those claims go through unchecked, and the record says so. A site can hand the plugin a list of its own for those bots, or treat every claim it can’t check as a fake.
The check works for bots outside the AI world too. Scrapers have posed as Googlebot for years to get past limits. Two lines in the plugin’s config add Googlebot with Google’s own address list, and the fakes are turned away while the real Googlebot carries on.
What It Tells You
Every bot request leaves a note in BareProxy’s record, so why explains it:
GPTBot (OpenAI, training): 20.171.206.5 is in OpenAI's list; let in by line 2, its 41st request let in this hour
fake GPTBot: 203.0.113.9 isn't in OpenAI's list (18 ranges); turned away (fake block)
A report page adds it up for each site: every bot’s requests today and yesterday, split into let in, limited, blocked and fakes, and the parts of the site each bot read. Bots that get in carry two headers, X-Bot and X-Bot-Check, so the app behind BareProxy knows a verified bot when it sees one. In the tests, the plugin spent about 0.07 milliseconds on an average bot request.
Next
Plugin 11 is bot challenges, a proof-of-work check for clients that look like scrapers, with no outside service. Two later plugins build on today’s. The AI crawler payment gate will quote prices to crawlers that pass the check, and RenderCache will give admitted bots a fully rendered page. The full order is on the roadmap.