System requirements

Functional:

1. receive various payment events from a payment gateway. Event types may be:
2. payment success
3. payment failure
4. refund requested
5. refund issued
6. chargeback
7. analyze payment event types and trigger corresponding actions

Non-Functional:

1. security. We don't want any random client access any random webhook APIs
2. strong consistency because of payments being involved
3. high availability. However, in edge cases, consistency takes precedence

Capacity estimation

Assume 100 million events daily.

Write / read QPS = 100M / 86400 = 1.2k QPS

Use 2x at peak, 2.4k QPS at peak

Assume each incoming event is 1 KB. Inbound throughput = 1.2k * 1 KB = 1.2 MB

Assume each event triggers an action of 1 KB. Outbound throughput = 1.2 MB

Throughput can be fit on one single server. QPS can also be handled by one server. However, users may come from different regions globally, and for availability, we may want sharding and replication.

API design

Incoming events:

POST /webhooks/events. Request contains an event payload and include a secret auth token

Database design

incoming events:

100M * 365 * 5 yr * 1 KB = 183 TB for 5 yrs. Assume some increase per year, say 250 TB for 5 yrs

outgoing events:

similar to incoming, say 250 TB for 5 yrs

Typically too big for a RDB. However, since a webhook event is always tied to an auth token which is linked to a specific account, we can shard RDB based on accounts. Due to strong consistency requirement, we choose RDB.

High-level design

See high level diagram

Request flows

1. client sends an event
2. server records the event to DB
3. server validates auth secret. If failed, server reject with error.
4. server checks event type and sends out corresponding triggered event out.
5. server records outgoing event to DB

Detailed component design

1. We need a rate limiter on server side to prevent user abusing the service. There're user specified rate limit (for themselves to prevent abusing by accident) and global rate limit (for our service to reject attackers)
2. Use load balancer to route incoming events to different server replicas based on user account / auth secret
3. If service detects many rejections of auth token from some client, service will block access. This could be a tiered blocking, starting with temporary block, and to permanent block per multiple violations. Block may happen at both client IP address level as well as account level
4. DB is shared by user account / auth secret
5. Use replicas for both servers and DBs. Use leader-follower setup. At a time, only one leader is elected and respond to incoming events. Leader may distribute triggering logic to followers: after a follower is kept up-to-date of a leader, the follower will trigger an outgoing event. As webhook events are triggered async and latency is not critical in most cases, we choose strong consistency and this flow works well.

Trade offs/Tech choices

We have chosen strong consistency because of financial events being involved.

We used RDB and shared by user accounts because events from different user accounts do not share overlap information between each other.

We use replicas for high availability.

Failure scenarios/bottlenecks

When a leader is dead, the followers will elect a new leader, e.g., use Raft algorithm. The new leader will be responsible for taking future incoming events.

Every leader and follower keeps a event log, so that in case they become dead, the new leader can recover from what was supposed to happen but not yet processed before the old leader went dead.

Future improvements

We may consider using NoSQL if consistency becomes less important in the future.