Design a Webhook Notification Service

Most systems you design in an interview are the caller, your service reaches out to databases, caches, third-party APIs. This problem inverts that model. Here, a payment gateway pushes events to you. You are the receiver. You do not control when events arrive, how many arrive at once, or whether the same event arrives twice. Your job is to accept every event, verify it is legitimate, process it exactly once, and never lose one.

This mental shift matters because it changes what you optimize for. A typical service optimizes for how fast it can fetch data. A webhook receiver optimizes for how fast it can acknowledge receipt, return 200 OK: before the sender gives up and retries.

Functional

Traffic

A large merchant on a payment platform like Stripe might process 10,000 to 100,000 transactions per day. Each transaction generates at least one webhook event (payment_success or payment_failure), and some generate additional events (refund, chargeback). Assume 100,000 webhook events per day at steady state.

Spread evenly: 100,000 events / 86,400 seconds = roughly 1.2 events per second on average. This is trivially low throughput.

Inbound Webhook Endpoint

POST /api/webhooks/events

This is the single endpoint that the payment gateway calls. The gateway sends event data in the request body and a cryptographic signature in the header.

Events Table (PostgreSQL)

1CREATE TABLE events (
2    event_id       VARCHAR(64) PRIMARY KEY,
3    event_type     VARCHAR(64) NOT NULL,
4    payment_id     VARCHAR(64) NOT NULL,
5    amount         BIGINT,
6    currency       VARCHAR(3),
7    status         VARCHAR(20) NOT NULL DEFAULT 'pending',
8    raw_payload    JSONB NOT NULL,
9    signature      TEXT NOT NULL,
10    retry_count    INTEGER NOT NULL DEFAULT 0,
11    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
12    processed_at   TIMESTAMPTZ,
13    updated_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
14);
15
16CREATE INDEX idx_events_payment_id ON events(payment_id);
17CREATE INDEX idx_events_status ON events(status);
18CREATE INDEX idx_events_created_at ON events(created_at);

The event_id is the primary key and comes directly from the payment gateway (e.g., evt_1NqFnLD4gKFsHx). This is the natural idempotency key. Using INSERT ... ON CONFLICT (event_id) DO NOTHING handles both the happy path (new event, insert succeeds) and the duplicate path (retry, insert is silently skipped) in one atomic statement.

The architecture separates two concerns that must operate on different timescales: fast acknowledgment (milliseconds) and reliable processing (seconds to minutes).

Payment Gateway sends webhook events to the Load Balancer, which distributes requests across multiple Webhook Receiver instances. The receiver is the fast path: it validates the HMAC signature, checks Redis for duplicates, stores the event in PostgreSQL as pending, enqueues it to Kafka, and returns 200 OK. Total time: under 100ms.

Webhook Receipt and Acknowledgment Flow

This is the synchronous fast path. every step must complete within the gateway's timeout window.

This section deep-dives into the components that make the webhook service process events reliably. The core challenge, and the GATE topic for this problem, is HMAC signature verification: ensuring that every incoming webhook is cryptographically authenticated before any processing occurs.

HMAC Signature Verification (GATE)

Synchronous vs Asynchronous Processing

Synchronous: Validate signature, process event, update database, return 200 OK. All in one HTTP request. Simpler code, atomic behavior, no message queue needed. But if processing takes more than the gateway's timeout window (5-30 seconds), the gateway retries. If a downstream service is slow, every webhook backs up.

Asynchronous (this design): Validate, store, enqueue, return 200 OK immediately. Process later via Kafka consumer. More complex (two components, a message queue, retry logic) but eliminates the timeout problem entirely. The receiver never blocks on downstream services.

Webhook Receiver Downtime

The receiver goes offline (deployment, crash, infrastructure failure). The payment gateway sends events but receives no response. It retries according to its schedule: typically 3-5 attempts over 24-72 hours with exponential backoff.

Recovery: When the receiver comes back online, retried events arrive. The idempotency mechanism handles any events that were partially processed before the crash. Events received during recovery are processed normally. If the outage exceeded the gateway's retry window, some events may be permanently lost: this is why 99.9% uptime is a hard requirement, not a nice-to-have.

Multi-Provider Webhook Support

The current design handles a single payment gateway. Extending to multiple providers (Stripe, PayPal, Adyen) requires a plugin-based validation architecture. Each provider has its own payload format, signature scheme, and retry behavior.

A WebhookValidator interface defines verify_signature(request) and parse_payload(request). Each provider implements this interface: StripeValidator, PayPalValidator, AdyenValidator. The receiver identifies the provider from the URL path (/webhooks/stripe, /webhooks/paypal) or a header, loads the appropriate validator, and delegates verification and parsing.