# System Design Interview: Designing a Webhook Delivery System Platforms like Stripe and GitHub notify customers by POSTing events to URLs the customer registers. Designing that reliably is a good interview question. ## Requirements - Customers register endpoints and choose event types. - Every event is delivered at least once, even if the customer's server is down for hours. - One slow or broken endpoint must not delay everyone else. ## Flow 1. When a business event happens, the producing service writes it to an **outbox** table in the same transaction as its own change, so events are never lost. 2. A relay publishes events to a queue. A fan-out service looks up subscriptions and creates one **delivery** record per (event, endpoint). 3. Delivery workers POST the payload with a timeout (say 10 s). A 2xx means done; anything else schedules a retry. ## Retries Exponential backoff with jitter (1 min, 5 min, 30 min, 2 h ... up to ~3 days), then mark the delivery failed and alert the customer. Store attempts so customers can see history and trigger a manual redelivery. ## Isolation Keep a queue (or a rate-limited lane) **per endpoint** or hash endpoints into many queues, and cap concurrency per endpoint. Otherwise one endpoint that hangs for 10 s per request ties up all workers. Disable endpoints that fail for days. ## Security - Sign each payload with an HMAC of the body and timestamp using a per-endpoint secret; receivers verify the signature and reject old timestamps to stop replays. - Resolve and block private IP ranges before connecting, to avoid SSRF into your own network. ## Receiver expectations Document that delivery is at-least-once and possibly out of order: include an event ID for dedup and a timestamp or version so receivers can ignore stale updates. --- *Disclosure: I work on [Linkjob AI](https://www.linkjob.ai/), an AI job interview assistant.*