BareProxy Adds a Webhook Inbox That Keeps Every Event Until the App Takes It

BareProxy 0.16.0 is out with a webhook inbox. It takes webhooks from Stripe, GitHub, Shopify and anything else that sends them, writes each one to disk, and then hands it to your app. If the app is down, the inbox keeps trying until the app is back.

Webhooks have a weak spot. The sender tries a few times and then gives up. An app that is being deployed, or has crashed, or is simply slow for ten minutes, loses whatever arrived in that window. Most teams find out when a customer asks why a paid invoice still shows as unpaid.

The proxy is already the thing every webhook passes through, so it is a good place to catch them.

How It Works

A route and an inbox block in the config:

site hooks.example.com
  route /hooks/stripe -> inbox stripe

inbox stripe
  deliver app
  signature stripe /etc/bareproxy/stripe-secret
  key-header Idempotency-Key

pool app
  backend 127.0.0.1:8080

A webhook that comes in on /hooks/stripe is checked against Stripe’s signature first. A bad one gets 401 and is thrown away. A good one is written to disk and flushed, and only then does Stripe get 202 Accepted. The disk write comes before the yes on purpose. If BareProxy can’t store the event, Stripe gets 503 and keeps it, the same as it would for any failed delivery.

Then the inbox delivers it to the app, with the same method, path, headers and body. If the app answers with a 2xx, that event is done. Anything else, or no answer, means another try later: after 10 seconds, then 20, then 40, doubling up to an hour between tries. After three days without success the event is marked failed. Both numbers can be changed.

Signatures work for GitHub, Shopify and Stripe by name, and for any other sender that signs the body with HMAC-SHA256. key-header names the header where the sender puts its own event ID. When the sender retries an event the inbox already has, it gets the same answer back and nothing is stored twice.

No Event Lost, None Sent Twice Without a Mark

The hard part of an inbox is the moment between “sent to the app” and “the app said yes”. If BareProxy dies right there, it can’t know whether the app got the event.

So before each attempt the inbox marks the event as in delivery on disk. When BareProxy starts and finds an event still marked that way, it was cut off in the middle. The event goes out again with a header that says so:

BareProxy-Inbox-Id: 18e3927c0bde516f5bdad3
BareProxy-Inbox-Attempt: 2
BareProxy-Inbox-Replay: in-doubt

The same mark goes on a retry after a timeout, where the request left but no answer came. A retry after a refused connection gets no mark, because the app never saw anything. An event the app accepted is never sent again unless someone replays it by hand, and then the header says operator. An app that treats marked deliveries with care, by checking the event ID it already has, sees every event exactly once.

Listing and Replaying

The inbox keeps events for 30 days by default. Three commands talk to the running server:

bareproxy inbox list stripe --status failed
bareproxy inbox show stripe 18e3927c0bde516f5bdad3 --body
bareproxy inbox replay stripe --failed

list shows each event with its last result and when the next try is due. show prints the request and every attempt. replay sends events again. After a bad deploy that rejected a day of webhooks, one command sends all of them back once the fix is live.

Each webhook also leaves a request record like any other, so why shows which inbox took it and the event ID it got. explain shows what an inbox route would do, signature check included.

How It Was Tested

The main promise is that nothing is lost, so the main test kills BareProxy on purpose. It runs BareProxy as a real process, sends it 300 webhooks, and kills it with SIGKILL while deliveries to the app are in flight. Three times in a row, then one more start. A sender that got no answer during a kill sends again, as a real one would.

The test ran five times. Across 1,500 webhooks and 15 kills, the app got 1,548 deliveries. None of the 1,500 events was lost. None reached the app twice without a mark. The 48 extra deliveries were all marked in-doubt, one for each attempt a kill cut off.

Another test stores 20 webhooks while the app is down. Every one got 202, every one was delivered once after the app came back, and none was marked, since no connection had been opened. A third checks that the waits between tries follow the backoff. The rest cover each operation: the four kinds of signature, good and bad, the size limit, the sender’s event ID, replays, giving up, and an inbox staying open across a config change. All of them pass, and the logs are in results/inbox-test.log.

Why It Lives in the Core

Most new BareProxy features arrive as plugins, small WebAssembly programs in a sandbox. The inbox couldn’t be one. It has to flush to disk before it answers, and a plugin can’t. It has to call the app on its own clock, across restarts, and a plugin only gets to run while a request or a timer tick is in progress. And replay is a command an operator types, which plugins don’t have.

So the inbox is part of BareProxy itself, written in Go like the rest of the kernel. Deliveries go through the same backend pools as normal traffic, so they share health checks and connection counts.

Limits

It is new. It has been tested on one machine against a local app, not yet against real traffic from Stripe or GitHub. One BareProxy should own an inbox folder; two sharing one would deliver events twice. Stored events include the headers as they came, so keep the state folder private. The inbox manual has every setting.

Three case studies run the inbox through an app deploy, a crash in the middle of deliveries and repeated events and a bad deploy, and the use examples post has short recipes.

The same release adds a DNS and TLS drift watcher. Both are on the plugins page under what’s in the kernel, and the binaries are on the releases page.