BareProxy Adds Bot Challenges That Make Scrapers Posing as Browsers Solve a Puzzle First
BareProxy 0.13.0 is out with its eleventh plugin, bot challenges. It’s the second release today. The first, AI crawler control, handles the bots that say who they are. This one is for the ones that don’t.
Those are the scrapers that send a browser’s user agent and spread their requests over thousands of addresses. Rules about user agents don’t catch them, since they look like browsers, and neither do rate limits, since no single address sends much. Small sites and code forges have been knocked offline by them.
The plugin is challenges.wasm on the releases page. Its page on this site has the details.
How It Works
A client that says it’s a browser, and has no pass yet, gets a small page instead of the site. The page holds a puzzle, and the browser’s JavaScript solves it: it looks for a number that, hashed with SHA-256 together with the puzzle, gives a result starting with 16 zero bits. That’s about 65,000 tries. A desktop browser does it in a fraction of a second.
The answer goes back to BareProxy, which checks it with a single hash and hands out a pass: a signed cookie, tied to that browser and its network, good for a week. Then the browser lands on the page it asked for.
Most scrapers that pretend to be browsers never run JavaScript, so they never get past the puzzle page. The ones that do run it pay for every pass in computing time.
The plugin doesn’t store puzzles or passes. Each one carries its own signature, so the plugin can check it without keeping a copy. No outside service is involved either.
Who Gets the Puzzle
By default, only clients that claim to be browsers. A bot that names itself goes straight through, Googlebot among them, and so do tools such as curl and anyone asking for robots.txt. Those are jobs for AI crawler control and robots.txt. A site can loosen or tighten the rest in a few lines:
after 10 per 1m
always hosting.txt
trust-header X-Bot-Check=verified
The first line lets a visitor load ten pages a minute before the puzzle appears. The second puts the puzzle in front of the networks listed in hosting.txt right away, such as hosting providers’ ranges, since people rarely browse from there. The third lets the bots AI crawler control has checked go on without a puzzle.
Tested in Chromium
The page is what visitors see, so it was tested in a real browser. Headless Chromium asked for a page, got the puzzle, solved it with the page’s own code, got its pass and was on the real page 0.14 seconds after asking for it. A reload went straight through. With JavaScript turned off, the page said the puzzle needs it. The script for that check is in the code repository, so anyone can run it again.
What It Costs Visitors
A moment, once a week, on a desktop; a little longer on an older phone. Visitors without JavaScript can’t pass at all, which is why the plugin can wait for a few page loads before it asks. A scraper that drives a real browser can still get through. The puzzle only makes each pass cost it time.
Next
Plugin 12 is Markdown serving: Markdown files turned into web pages straight from a folder, with no build step. After it come cookie consent, status pages, OpenTelemetry export and analytics. The full order is on the roadmap.