Loading...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST /api/webhook/events(event id,type,paymentid,amount, currency, status,timestamp and signature) respone 200k then asycn
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Let's start with a simple flow on a happy path. A client will talk to payment gateway, and there it will create an idempotent key, which will be a user's token plus a UUID. And that key will be passed along with the payment information with the hook. And the hook will first see if this key has already been processed. If it's already been processed, it will actually store, get the stored response from the Redis cache. If not, it will process it and request the response from it. The hook will immediately respond back with HTTP status 202 OK, and then it will send the event to event bus. In the event bus, we will have a different consumer called payment processor. It will consume the data and it will try to then talk to, like, the acquirer, the card issuer, and then it will get that information if the particular payment has been processed or not. Here, we will actually use the retry mechanism with the card issuer if suppose anything is blocked, we will retry with the same token ID. We'll create a token also in the hook and we will have the retry mechanism. If after certain amount of retries it's not getting fulfilled, then we will put that into a letter queue.
For event logging, we will have, because this will be a high processing transaction, so if we need something, we are log or needed, we will actually use the Kafka, and we will partition this based on per user. And for every, so that inside the inside the partitions of the topic, we can actually know what the offset is and we can replay the events and kind of lock it for any future references. We can also have an analytic DB which can actually take care of this. Now, second thing is that regarding the HTTP status 200 OK, in the event bus, we will have our established backpressure queue so that we can have a certain limits, and after that limit is done, we will just say that the event is down, so we'll also send that. And because we have focused on 10,000 requests, so I think we should have that into our mind, and we will create those things into which.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
so for the item put into the key, I think the client will generate the key and the key will have the user ID as well as the UUID version 4, because it's cryptography enabled. And with this key, we will store this key into the Redis cache along with the response. If the response is 202 OK, we will send that to the user. And then the user can, once the webhook event arrives, then those things can be processed independently via the notification service and all those things. Now, with the webhook integration, we will also have a HMAC signature, so we will also check if the signature is valid. And if the signature is not valid, we will not request the webhook. Regarding like handling concurrency, concurrent requests, what we are going to do is, as I've already mentioned, we will be having a support as well as we will be having a rate limiting in the payment gateway site. If there are a lot of requests, we will be trying to use a backpressure queue along with the hard-coded like retry or timeout mechanism, like this much quantities we can actually preserve. And this will be it.
See, retry mechanism will be done via a simple task where we will use exponential backoff as well as jitter, so that suppose for if the payment, like if the acquirer or card issuer server is down, we should, and there are 100,000 or 10,000 concurrent requests, not all of them should retry at the same time. So we will implement jitter where we will have certain, like a frame where we can do or not all requests are kind of hitting the acquirer and card issuer at the same point of time. And that is how we will be able to kind of store it. We will have like certain tools to power and some kind of exponential backoff. Regarding the HMAC integration, we will have that signed key one with us, and we will have a signed signature key from public to private API keys, and that will be done.
Now, second thing is Redis cache, where suppose we are coming with the cache miss problem. See, for cache miss problem, we can have two things. We will first use the request coalescing here, so that if suppose there are a lot of concurrent requests coming for the same, like, database or same event, we will just first go with one event and kind of come back with it, and we'll be able to handle that. And at the same point of time, like for a very hot request, we will try to replicate the sharding on multiple instances rather than on the one instance having that hotkey. And the HMAC instance, as I already mentioned, it happens like where, like sender will hash the payload with a secret, shared secret, and it will include it in a header. And then the receiver will recompute the HMAC using the same secret and compare it. If the match verifies, authentication and integrity is maintained, otherwise we will log off.If we partition by user, we will be able to kind of catch it. We can also partition by user and payment type later on the line, but yeah.
Webhook receiver will be deployed in a containerized environment like Kubernetes, and then we will have horizontal pod scaler based on the CPU memory or custom metrics. During a spikes, we can additional pod will be automatically spun up, and during low traffic, it's not. Load balancing will be like incoming web requests will go through a layer 7 load balancer, like Nginx or AWS load balancer, and it will distribute the requests across the pod. Sticky session is not required here, so rate limiting and load balancing ensure that there will be no single instance will be overwhelmed. Okay, so on a combined effect, you can see the auto scaler will add the capacity, the pod where the load balancer will ensure the event traffic, and we can retry with exponential backoff and GTN2 to prevent the storm of hitting all instances simultaneously.
circuit breaker
circuit breaker will keep the connection open if it finds out that everything is failed and then after some time it will keep the connection half open, send one request, see if everything goes well, then it will say the 10% of it, if not, then it will keep the connection open. 10% also passes, then only it will say that the connection is closed.
Use Kafka only for events requiring reliability, ordering, and replay; use simpler mechanisms elsewhere to reduce operational complexity and latency