How the BareProxy Inbox Kept Webhooks Through a One Minute App Deploy

Most webhook losses happen during deploys. The old process stops, the new one takes a minute to come up, and whatever Stripe or GitHub sent in that minute gets an error. The sender retries a few times on its own schedule and then gives up.

This case puts the webhook inbox between three senders and an app, stops the app for about a minute and keeps sending. It ran on the released BareProxy 0.16.0 binary for Linux. The app and the senders are small Python scripts on 127.0.0.1. The webhooks use the real formats and signatures of GitHub, Stripe and Shopify, signed with secrets made fresh for the run, and none of those services was contacted.

The setup

Three inboxes, one per sender, all delivering to the same app. The retry settings are the defaults: 10 seconds after the first failure, doubling each time, up to an hour between tries.

site http://hooks.test:36707
  route /hooks/github -> inbox github
  route /hooks/stripe -> inbox stripe
  route /hooks/shopify -> inbox shopify

inbox github
  deliver app
  signature github /work/github-secret
  key-header X-GitHub-Delivery

inbox stripe
  deliver app
  signature stripe /work/stripe-secret

inbox shopify
  deliver app
  signature shopify /work/shopify-secret
  key-header X-Shopify-Webhook-Id

pool app
  backend 127.0.0.1:40513

The app writes a line for every delivery it gets and remembers the inbox IDs it has handled, so it can tell a repeat from a new event.

The deploy

Three webhooks went through while the app was up and reached it within a tenth of a second. Then the app was stopped. For the next 67 seconds a webhook came in every 4 to 5 seconds, 15 in all, rotating between the three senders.

Every one of them got 202 and an inbox ID. The median time to that answer was 59 ms and the slowest took 153 ms, and that includes writing the event to disk and flushing it before answering. From the sender’s side nothing was wrong.

Halfway through the wait, the GitHub inbox looked like this:

$ bareproxy inbox list github
Inbox github (line 11), to pool app, in /work/state/inbox/github
  5 pending, 0 delivering, 1 delivered, 0 failed
  18e39732bc9e6d9b2b755a  10:42:00  pending    1 attempt, last: no backend is up, next in 5s
  18e3972fa37a4e1327c3f1  10:41:47  pending    2 attempts, last: no backend is up, next in 12s
  18e3972c807b91ec9930e9  10:41:34  pending    3 attempts, last: no backend is up, next in 39s
  18e3972975e2c373362c45  10:41:20  pending    3 attempts, last: connect refused, next in 26s
  18e397265e8eaa1c17badf  10:41:07  pending    3 attempts, last: no backend is up, next in 12s
  18e397239ca0962237dd6f  10:40:55  delivered  1 attempt, last: 200

(Columns trimmed to fit; the full output is in the log.) Two kinds of failure show up. “Connect refused” is a real attempt that found nothing listening. “No backend is up” is the pool saying it already knows the app is down, so the inbox doesn’t bother opening a connection. Neither counts as in doubt, since the app never saw a request.

After the app came back

The app was started again at 70.3 seconds. The first waiting event reached it 7.7 seconds later. The rest arrived as their retry timers came due, and the last one landed at 153.7 seconds. That was the first Stripe event sent during the outage, which had been waiting 150 seconds and went through on its fifth attempt.

The totals from the run:

Attempts per event: three took one try (sent before the deploy), five took three, nine took four, one took five.

What it shows, and what to tune

Nothing was lost, and the app saw each event once without having to dedupe anything. That is the main promise and it held.

The run also shows the cost of the default backoff. A 67-second outage turned into a two and a half minute tail of late deliveries, because an event that has already failed three times waits 40 seconds before its fourth try, however soon the app comes back. For an app that deploys often, a shorter ceiling such as retry 5s 2m keeps that tail short. The defaults are tuned for long outages, where an hour between tries is kind to a struggling app.

Events also arrive out of order after an outage. Each one keeps its own timer, so a webhook sent at second 48 reached the app before one sent at second 4. An app that cares about order should look at timestamps in the payload, which Stripe, GitHub and Shopify all include.

The script and the full log are in the repository: inbox-deploy.sh and inbox-deploy.log. The use examples post has the config for a deploy-prone app, and the next case study kills BareProxy itself in the middle of deliveries.