High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

Okay, so the client comes to the API gateway and we talk to a task service. The purpose of task service is to insert a particular request into a database and then the scheduler service takes that particular position. If it's a single time, it will take the single time. If it's a recurring time, it will take that. At any point of time, if you want to do something, the scheduler service will take into picture. It will have an in-memory priority queue along with the write-ahead log so that anything can be handled. Then it will put that into the message queue, which will be a Kafka. And then task executor will try to consume whatever messages are there and it will try to update. And even if multiple retries, the task is not scheduled and it will go to the DLQ. And once the task is completed, it will update the database table that, okay, this task is completed and as well as a notification service has been triggered.

we can keep the API gateway as a routing mechanism, where it will route, authenticate, and also take the rate limiting part, and then it will actually send the one thing to task service. The sole purpose of task service is to put the information in the database, and it will just put all the information like when is the start task single time or recurring time, and all those things. For the recurring time or single time, then we will take the scheduler. The scheduler will scan for the task, and it will take up all the chronicle job, and in that case, all the chronicle jobs will be put into a priority queue, based on the priority of the jobs, it will then send it to the message queue. The message queue will be based on partitions, like you will have partitions, and then there will be consumer groups, like task executor, who will be taking it. The task executor, you can think of it like there will be consumer, which will be running for separate, separate partitions and separate, separate works, and they will be running for it. So this is how they will be executing it. And once they are executed, they are giving down, like they will send a notification.

Interesting approach! How will you ensure that the system can handle bursts of tasks effectively?For a burst of tasks, we will have a, in the tasks in a scheduler service, we can respawn the scheduler with the help of Kubernetes pods, and we can auto do the auto load balancer. And if there are a lot of tasks coming up, we will just respawn more pods and which can take more instances of scheduler, so which can actually take this up into the consideration. And for anything lost, we will actually maintain a write ahead log, and if anything goes on, we will get a new instance and then they will just take the write ahead log as a reference and kind of reprocess it.

One of the approaches we can take is from the Hikari, where we'll keep certain parts up and running, so that even on the spikes, we don't kill all the parts. So suppose like five parts are up and running, and in case any volume come, we can actually go on increasing the parts. We can have, in this way, we will be able to kind of have those tasks, scheduler instance always running, and so this will be a thing where we can, in certain spike, we will be able to at least have some of the scheduler free body.Regarding dead letter queue, the task executor will have each worker inside the task executor can be working on, actually you can say task executor is a consumer group which will consume the tasks from the Kafka message queue and it will retry and if the task is still failed, it will do the DLQ, it will set into a DLQ and which can later down, we can still go for manual review. So the task executor will try for three times with exponential backoff. If then also it fails, it will keep that circuit as break and it will keep on that. Also, we will keep circuit breaker for any task execution. So suppose if certain tasks are failing after a certain time, so we can just say that, okay, we are not passing it out.

Submissions

Displaying Solutions with Scores Ranging from 1 to 10