Requirements

Functional Requirements:

• Receive webhook events from the payment gateway for payment successes, failures, and chargebacks.
• Validate webhook authenticity using signatures or token-based authentication, including HMAC signature verification.
• Process events idempotently using idempotency keys, ensuring that repeated calls with the same event ID do not lead to inconsistent state changes.
• Automatically retry processing events in case of transient failures using an exponential backoff strategy with a maximum retry limit.
• Persist and log received events and their statuses (success, failure) for auditing and troubleshooting.

Non-Functional Requirements:

• We should be able to deliver the events at-least-once
• We should not block the main operations in case of errors while logging events. The database will be the main source of truth.
• We need to generate beforehand a secret for HMAC to be distributed with the receivers of the webhook.
• Events do not need to follow an order

API Design

POST /event: Stores an event in the database generating an id to be used as idempotency key after performing the hmac validation. For now we will the typical sequential primary key from an SQL database. If it scales we can think of a ID generator with snowflake IDs.

GET /event/{id}: Returns the event by ID with its type (success, failure, chargebacks), webhook status (success, failure) and hmac validation.

High-Level Design

1. The payment gateway sends an event to the system via API.
2. The API servers validate that the request is legit using HMAC and store the event in the database, generating an ID, and writing a log with the operation.
3. The API servers publish a message in the queue (SQS) to be consumed by the workers. The workers use the event ID generated when storing the event as idempotency key to identify whether that even has been seen before. If that is the case, they do nothing, otherwise they execute the next step. They log all these operations.
4. The workers call the webhook using the HMAC secret for authentication. In case of success update the database with status "success" for the webhook. In case of error, they retry using exponential backoff. If that fails too, the message is routed to a DLQ so it is not lost and the status of the event is marked as "failure". They log all these operations.
5. The reviewers consume the DLQ messages and trigger alerts.

Detailed Component Design

1. High availability and traffic handling: The load balancer is in charge of scaling up and down the API servers depending on the requests received. Workers also go up and down depending on traffic and usage. The logs can be leveraged by AWS Cloudwatch, otherwise we have the database records.
2. DB atomicity: We need to guarantee ACID transactions so we use pessimistic lock for the SQL instances (API servers and workers will be updating records, potentially the same). We will use sharding to increase write capacity in case we have lots of users. For reads, we will have replicas of the database.
3. We need to introduce an intermediate status "pending" to define when an event is written by the API server but not consumed just yet by a worker. This will help identify potential issues if all workers are down.
4. Both API servers and Workers write logs reflecting transitions. If for whatever reason we cannot write, it is non-blocking since the database will be reflecting the same information.