System requirements

Functional:

• Web-hook system where a sender can do a request
• Web-hook system where a receiver can process a request
• Web-hook system listens for payment events from payment gateway
• Web-hook system handles payment success, failures and chargeback events
• Scale - Mid scale i.e. 1000 to 10000 events day

Non-Functional:

• Securely receives and processes notifications
• Scalable horizontally
• Fault tolerance
• SLA of 99.999%. We expect downtime no more than 1 hours in an year.

Capacity estimation

• Considering 10,000 events per day = 10000/86400 = 1.5 events per second, we can round off that too 2 events per day (this will also include our buffer)
• Each event, contains a Payment State and Payment Metadata, Assuming it is 10KB
• 10KB * 2 = 20KB per second. This is the amount of Network Bandwidth we need
• 10K * 10K = 100MB Per day, 100 * 365 = 36.5GB per year approx, so, we need about 365GB storage for 10 years.
• SLA - 99.999%, Downtime per year is 52 mins, 52/12 = 4min downtime per month approx = 240/30 = 8sec per day approx acceptable downtime.

API design

POST /web_hook/v1/p_events

Input -

• String P_ID
• String P_STATE
• String URL

Output

• HTTP 200 OK
• HTTP 4XX - Invalid Requests
• HTTP 5XX - Web-hook Service Errors

Database design

Transaction Table - Since, this is a payment system we need strong consistency and ACID guarantees. Also, that our load is significantly lower than 5 TB. So, we choose RDBS to store transaction data

Transaction Table

string transaction_id

string customer_id

string merchant_id

bigint amount

We use transaction table to validate transactions for each action.

High-level design

Request flows

• Payment gateway send payment events
• We use DNS based geo-routing to route to the nearest datacenter
• From there request is sent to nearest API Gateway
• web_hook service next performs validation for the transaction. To expedite this it reads through the cache, if cache is missed, it reads through Database and updates the cache. If successful we proceed further else return HTTP 400 or 422 (Unprocessable entity) depending on the error.
• web_hook service next puts the event to the messaging queue
• we have workers that read events from the queue and processes the request

Detailed component design

• While our messaging queue could ensure exactly once delivery. However, we know that in distributed system this is always dependent on the application. So, how do we ensure that we don't process a message more than once. To handle this we can implement a deduplication logic, we assign UUID to each message, we verify Message ID and process only if the id is not processed already.
• Workers, I am familiar with Rust. Rust does not have an async runtime and so, we rely on external libraries. Tokio is popular choice in the community, we can use Tokio Tasks and Channels. Task creates a task and channel provides us with Sender and receiver semantics
• To ensure multi datacenter setup we can start with two data-centers both active active and any data center can receive the write. If one is down we have route all traffic to one datacenter, the trade-off at this stage is that all requests to/from the other data center may be slowed down
• API Gateway, we use API Gateway as we want to ensure Security, we also want our service to efficient (avoid overload), we also want to load balance our requests. API Gateway handles all of this for us.
• key metrics to monitor
  • Uptime per day (SLA assurance)
  • Queue Size at a given instant of time (To ensure we have enough workers or we need to add more)
  • Latency at each layer, API Gateway, cache, Database, Message Queue
• Database: For transaction table we only need to read. Since, there could be other services writing to that table. We could use a read-only database also known as read replica.

Trade offs/Tech choices

• API Gateway vs Load Balancer. Since, we want to Authenticate users, ensure security and efficiency by using a rate limiter and we want to employ load balancing. Instead of using and managing individual services we employ an API Gateway to avoid operation overhead, while we are aware we may add a bit of latency (nothing significant though)

Failure scenarios/bottlenecks

• We rely on API Gateway. We must ensure it is distributed in nature

Future improvements