Engineering Note
Webhook Delivery & Reliability at Scale: Ensuring Nothing Gets Lost
"We sent the webhook."
The customer replied:
"We never received it."
That conversation taught us an important lesson about scaling integrations.
In a Rails SaaS platform, we were processing thousands of webhook events every hour.
Initially, the flow looked simple:
Event occurs → Send webhook → Done.
Until real-world failures started appearing.
* Customer endpoint was temporarily down
* Network timeouts occurred
* Third-party APIs responded slowly
* Some requests succeeded, but acknowledgements never returned
The scary part?
From our side, everything looked normal.
From the customer's side, data was missing.
So we changed the architecture.
Instead of treating webhooks as a request, we treated them as a delivery system.
What changed:
* Every webhook was persisted before delivery
* Sidekiq handled retries automatically
* Exponential backoff reduced retry storms
* Delivery status was tracked for every event
* Dead-letter queues captured failed deliveries
One customer endpoint was unavailable for nearly 2 hours.
After recovery, thousands of queued events were delivered successfully without manual intervention.
No lost data.
No support tickets asking for replays.
The important learning:
At scale, sending a webhook is easy.
Guaranteeing delivery is the real engineering problem.
Because customers don't care that the event was sent.
They care that it arrived.
The customer replied:
"We never received it."
That conversation taught us an important lesson about scaling integrations.
In a Rails SaaS platform, we were processing thousands of webhook events every hour.
Initially, the flow looked simple:
Event occurs → Send webhook → Done.
Until real-world failures started appearing.
* Customer endpoint was temporarily down
* Network timeouts occurred
* Third-party APIs responded slowly
* Some requests succeeded, but acknowledgements never returned
The scary part?
From our side, everything looked normal.
From the customer's side, data was missing.
So we changed the architecture.
Instead of treating webhooks as a request, we treated them as a delivery system.
What changed:
* Every webhook was persisted before delivery
* Sidekiq handled retries automatically
* Exponential backoff reduced retry storms
* Delivery status was tracked for every event
* Dead-letter queues captured failed deliveries
One customer endpoint was unavailable for nearly 2 hours.
After recovery, thousands of queued events were delivered successfully without manual intervention.
No lost data.
No support tickets asking for replays.
The important learning:
At scale, sending a webhook is easy.
Guaranteeing delivery is the real engineering problem.
Because customers don't care that the event was sent.
They care that it arrived.