plugins

BareProxy Bot Challenges Plugin: A Proof-of-Work Check Before Scrapers Reach the Site

Status: planned, number 11 of 21 in BareProxy’s build order. The plugins are built easiest first, and this one is a day or two of coding: a challenge page, a check and a signed cookie. It comes after the AI crawler control plugin. This post describes what it will do, and it will be updated as it is built.

Not every bot announces itself. Scrapers that ignore robots.txt usually send a browser’s user agent and spread their requests over thousands of addresses, so neither user-agent rules nor rate limits catch them. Small sites, code forges and wikis have been hit hard by this, sometimes to the point of going offline.

A challenge asks the visitor’s browser to prove it’s a browser before the site does any work. A real browser solves it in a moment, once, and gets a cookie for the rest of the visit. A scraper fetching a million pages either can’t solve it or has to pay for every solve in CPU time.

What It Will Do

  • Challenge only the suspicious. Requests that look like a browser but behave like a scraper, such as no cookies across many pages, or addresses from hosting providers, get the challenge. Known good bots and returning visitors don’t.
  • Proof of work. The challenge page runs a short computation in the browser and sends back the answer. The plugin checks it in microseconds. The difficulty is set in the plugin’s config.
  • A signed cookie. A solved challenge earns a cookie signed with a key only the plugin knows, valid for a time the config sets.
  • No outside service. Everything happens between BareProxy and the browser. Nothing about visitors is sent to anyone.

A config will look something like this:

plugin challenge /etc/bareproxy/plugins/bot-challenges.wasm
  config /etc/bareproxy/plugins/challenge.json

site git.example.com
  use crawlers challenge
  route /* -> forge

The Trade-Off

Challenges cost real visitors a moment, and visitors without JavaScript can’t pass one at all. That’s why the plugin challenges only what looks suspicious, and why it works best next to the AI crawler control plugin, which deals with bots that do say who they are.

Every challenge issued, passed or failed carries a note in the request’s record.

The whole program is on the plugins page.