BareProxy Bot Challenges Plugin: A Proof-of-Work Check Before Scrapers Reach the Site
Status: built, number 11 of 21 in BareProxy’s build order, and out in BareProxy 0.13.0 as challenges.wasm on the releases page. Its manual is the README in the code repository.
Not every bot announces itself. Scrapers that ignore robots.txt usually send a browser’s user agent and spread their requests over thousands of addresses, so neither user-agent rules nor rate limits catch them. Small sites, code forges and wikis have been hit hard by this, sometimes to the point of going offline.
A challenge asks the visitor’s browser to prove it’s a browser before the site does any work. A real browser solves it in a moment, once, and gets a pass for the rest of the week. A scraper fetching a million pages either can’t solve it or has to pay for every solve in CPU time.
What It Does
- Challenges what claims to be a browser. A client whose user agent says browser, and that has no pass, gets a small page with a puzzle instead of the site. Bots that say who they are go on, and so do tools such as curl, robots.txt, and the paths and addresses a site names.
- Proof of work. The page’s JavaScript looks for a number whose SHA-256 hash, together with the puzzle, starts with 16 zero bits. That’s about 65,000 tries, a fraction of a second on a desktop and a little longer on a phone. The plugin checks the answer with a single hash.
- A signed pass. A solved puzzle earns a cookie signed with a key only the plugin knows. The pass is tied to the browser and its network, and lasts a week unless the config says otherwise.
- Nothing kept about visitors. Puzzles and passes are signed, so the plugin keeps no record to check them against. Everything happens between BareProxy and the browser, with no outside service.
Here is a config:
plugin challenges /etc/bareproxy/plugins/challenges.wasm
config /etc/bareproxy/plugins/challenges.conf
store 1MB
site git.example.com
use crawlers challenges
route /* -> forge
And the plugin’s own, which can be empty:
# challenges.conf
after 10 per 1m
always hosting.txt
trust-header X-Bot-Check=verified
after lets a visitor load ten pages a minute before the puzzle appears. always names networks, such as hosting providers’, that get it at once, since people rarely browse from them and scrapers often run there. trust-header lets the bots that the AI crawler control plugin has verified go on without a puzzle. store keeps the key the plugin makes for itself, so passes survive a restart.
Tested in a Real Browser
The puzzle page is the part a visitor sees, so it’s tested where visitors are. A script loads a page through the plugin in headless Chromium: the browser gets the puzzle, solves it with the page’s own code, gets its pass and lands on the page, picture and all, well under a second after asking for it. With JavaScript turned off, the page says the puzzle needs it. The plugin’s other tests solve puzzles in Go and try forged passes, old answers and answers solved for someone else.
The Trade-Off
Challenges cost real visitors a moment, and visitors without JavaScript can’t pass one at all. That’s why the plugin can wait until an address has loaded a few pages, and why it works best next to AI crawler control, which deals with the bots that do say who they are. A scraper that drives a real browser can still solve the puzzle; the work makes every solve cost it time.
Every challenge issued, passed or failed carries a note in the request’s record.
The whole program is on the plugins page.