System requirements

Functional:

• users can register events in the system
• when some events occur, they triggers coresponding actions registered

Non-Functional:

• high scalability
• high availablity
• reliability: actions should be executed at least once
• security between the wekhook system and the endpoints

Capacity estimation

Estimate the scale of the system you are going to design...

API design

Database design

webhooks

• webook_id
• created_at
• secrets
• event_type

webhook_request

• webhook_request_id
• webhook_id
• http_status
• retry_num
• next_retry_num

High-level design

webhook management service: CRUD webhooks

webhook controller:

• create webhook requests, add that into webhook devlivery log datastore
• add the webhook request into the message queue

Workers pull the webhook requests from the MQ and call downstream

worker write the status into the webhook delivery log datastore

Request flows

Addressed in high-level design

Detailed component design

Reliability

We will gurantee at-least delivery by retrying the requests. expoential retry

Security

authn: We can ask users to provide HMAC security so that it can be used as authn

payload encryption: Use asymmterical encryption/symmetrical encription. exchange secret keys by RSA

Availability

Webhook metadata datastore: non-relational/relational DB is fine for either, we don't need cache suppose

Webhook delivery log datastore: non-relational DB, it is write-heavy. we can shard by request_id

Message queue: shard by request_id

Scalablity

We can add more machines for webhook controller/webhook management service/webhook controller-cron as they are stateless components

As for

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

• add LB to the webhook controller to prevent traffic spikes