plugins

BareProxy AI Crawler Payment Gate Plugin: 402 Payment Required for Bots That Want the Content

Status: planned, number 19 of 21 in BareProxy’s build order. The plugins are built easiest first, and this one is about four or five days of coding: prices, proof of payment and books, on payment schemes still settling. It comes after the Response cache plugin. This post describes what it will do, and it will be updated as it is built.

AI companies need fresh, well-written pages, and a growing number of publishers want to be paid for them. The web has had a status code for this since 1997 that almost nobody used: 402 Payment Required. It is finally getting a job. A site answers a crawler with 402 and its terms; the crawler pays and comes back with proof; the site serves the page.

The payment gate puts that exchange at the proxy, so a site can price its content for bots without touching its code.

What It Will Do

  • Price by bot and by path. One rate for a training crawler, another for a search assistant fetching a single page to answer a question, nothing for the site’s own sitemap. Free for some bots, closed to others.
  • Answer 402 with terms. The response says what a page costs and how to pay, in headers a crawler can read without parsing HTML.
  • Check proof of payment. A paying crawler sends a signed token. The plugin checks it, either on its own against a key in its config or by calling the payment provider at an address its config allows.
  • Keep the books. Every paid request is counted by bot, path and price in the plugin’s store, so the bill and the logs agree.

It works with the AI crawler control plugin, which tells real crawlers from impostors before the gate quotes them a price. A config will look something like this:

plugin crawlers /etc/bareproxy/plugins/ai-crawler-control.wasm
  config /etc/bareproxy/plugins/crawlers.json
plugin paygate /etc/bareproxy/plugins/payment-gate.wasm
  config /etc/bareproxy/plugins/prices.json
  store 256MB

site example.com
  use crawlers paygate
  route /* -> files /var/www/example/public

Open Questions

There is no single standard for how a crawler pays yet. Several schemes are being tried, from payment networks run by CDNs to signed tokens and micropayment protocols. That’s one reason it sits late in the build order: by then one or two schemes should have clearly won. And because it’s a plugin, a new scheme is a new plugin release, never a change to BareProxy itself.

The whole program is on the plugins page.