Requirements

Functional Requirements:

• Receive webhook events from the payment gateway for payment successes, failures, and chargebacks.
• Validate webhook authenticity using signatures or token-based authentication, including HMAC signature verification.
• Process events idempotently using idempotency keys, ensuring that repeated calls with the same event ID do not lead to inconsistent state changes.
• Automatically retry processing events in case of transient failures using an exponential backoff strategy with a maximum retry limit.
• Persist and log received events and their statuses (success, failure) for auditing and troubleshooting.

Non-Functional Requirements:

• We should be able to deliver the events at-least-once
• We should not block the main operations in case of errors while logging events. The database will be the main source of truth.
• We need to generate beforehand a secret for HMAC to be distributed with the receivers of the webhook.
• Events do not need to follow an order
• We should return in less than 500ms to the payment gateway since we just create the record
• 99.9% availability
• Max 5 retries if errors (1,5,25,125,625 seconds)

API Design

POST /event: Stores an event in the database generating an id to be used as idempotency key after performing the hmac validation. For now we will the typical sequential primary key from an SQL database. If it scales we can think of an ID generator with snowflake IDs.

• Request: Payment status, webhook URL, HMAC authentication (header)
• Response: 201 (created) if success + event ID in the payload. 401 if request is not authorised or 500 if any other error happened when storing the record

GET /event/{id}: Returns the event by ID with its type (success, failure, chargebacks), webhook status (success, failure) and hmac validation.

• Request: ID in the URL + query parameters by payment status and webhook status (optional both of them).
• Response: 200 if success, 404 if not found, 500 if any error in the system. The payload returned consists on the payment status, webhook URL and its status.

POST /event/{id}/retry: Admin-only endpoint to trigger a manual retry for the webhook. It only works for webhook status "failure" and publishes in the queue. It takes no parameters and returns immediately 202 accepted.

Important: Cache is not needed at the beginning since the expectation is that the GET endpoint is not hit very often. If for whatever reason that changes, we can include a distributed cache whose eviction policy would be TTL (more likely that we want to know the status of recent events).

High-Level Design

1. The payment gateway sends an event to the system via API.
2. The API servers validate that the request is legit using HMAC and store the event in the database, generating an ID, and writing a log with the operation.
3. The API servers publish a message in the queue (SQS) to be consumed by the workers. The workers use the event ID generated when storing the event as idempotency key to identify whether that even has been seen before. If that is the case, they do nothing, otherwise they execute the next step. They log all these operations.
4. The workers call the webhook using the HMAC secret for authentication. In case of success update the database with status "success" for the webhook. In case of error, they retry using exponential backoff. If that fails too, the message is routed to a DLQ so it is not lost and the status of the event is marked as "failure". They log all these operations.
5. The reviewers consume the DLQ messages and trigger alerts.

Detailed Component Design

1. High availability and traffic handling: The load balancer is in charge of scaling up and down the API servers depending on the requests received. Workers also go up and down depending on traffic and usage. The logs can be leveraged by AWS Cloudwatch, otherwise we have the database records.
2. DB atomicity: We need to guarantee ACID transactions so we use pessimistic lock for the SQL instances (API servers and workers will be updating records, potentially the same). We will use sharding to increase write capacity in case we have lots of users. For reads, we will have replicas of the database.
3. We need to introduce an intermediate status "pending" to define when an event is written by the API server but not consumed just yet by a worker. This will help identify potential issues if all workers are down.
4. Both API servers and Workers write logs reflecting transitions. If for whatever reason we cannot write, it is non-blocking since the database will be reflecting the same information.