BareProxy Killed in the Middle of Webhook Deliveries and No Event Was Lost

The worst moment for a webhook relay to die is after it has sent an event to the app and before the app has answered. The app may have charged the card or shipped the order. The relay can’t know. If it sends the event again, the app might do the work twice. If it doesn’t, the event might be lost.

The inbox handles this by marking every attempt on disk before it starts. This case kills BareProxy with SIGKILL in exactly that window and looks at what happens next. It ran on the released 0.16.0 binary for Linux, on 127.0.0.1, with Stripe-format webhooks signed by a test secret made for the run.

The setup

One inbox, four deliveries at a time, and an app that takes 3 seconds over each webhook. That slowness is on purpose: it gives the kill a wide window to land in.

site http://hooks.test:35151
  route /hooks/stripe -> inbox stripe

inbox stripe
  deliver app
  signature stripe /work/stripe-secret
  workers 4

pool app
  backend 127.0.0.1:38011

The app keeps a file of the inbox IDs it has handled. A delivery whose ID is already in the file gets a 200 and is skipped. This is the idempotent handling the inbox manual asks of apps, and it takes about five lines.

The kill

Six webhooks came in within 1.5 seconds and each got 202. Four went straight to the app. The other two waited for a free worker. At 2.8 seconds BareProxy was killed with kill -9, no chance to clean up.

The state files it left behind tell the story. Four events read "status":"delivering" with one attempt started and no end. Two read "status":"pending".

The app didn’t know anything had happened. It finished the four webhooks it was holding between 3.5 and 4.2 seconds, processed them, and answered into a closed connection.

While BareProxy was down, a seventh webhook arrived. It got no answer at all, which is what a sender sees when any server is down. A real sender retries, so this one did too, 3.4 seconds later.

The restart

BareProxy came back at 6.1 seconds. On startup the inbox found the four events still marked as delivering, recorded the cut-off attempt, and sent them again within a tenth of a second, each carrying:

BareProxy-Inbox-Attempt: 2
BareProxy-Inbox-Replay: in-doubt

The app checked its file, found all four IDs, and skipped them. The two pending events went out unmarked, since no attempt had ever started for them. The seventh event, retried by its sender, was stored and delivered at 6.4 seconds. Everything was done half a second after the restart.

inbox show on one of the four says plainly what happened:

Event 18e39717fcbb1e9eb853af in inbox stripe: delivered
Received 2026-10-31 10:40:05.903 UTC: POST hooks.test:35151/hooks/stripe, 162 B
  ...
Attempt 1 at 2026-10-31 10:40:05: BareProxy stopped during the attempt, may have reached the app
Attempt 2 at 2026-10-31 10:40:11: 200 (marked in-doubt)
Delivered 2026-10-31 10:40:11 UTC

The count

So the app got every event and did the work for each exactly once. The four repeats were all marked, and the app needed only the inbox ID to spot them.

What an app has to do

The inbox is careful to say what it can’t know. In a crash like this one the app did get the first attempt, but BareProxy has no way to tell that from a crash a millisecond earlier. So it sends again and labels the delivery. The app’s part of the deal is to treat a labelled delivery as possibly seen. Keeping handled IDs in the same database transaction as the work itself is the usual way to do that.

An app that ignores the header would have processed four payments twice here. The marks make the problem visible. They don’t remove the need to handle it.

The script and the full log, state files included, are in the repository: inbox-crash.sh and inbox-crash.log. The earlier case study covers an app deploy, and the next one repeated events and a bad deploy.