Loading...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
post/task/create{task name,jobdetails, schedule type {once,cron},cron job? {},start date, end date} return 201 created
post/task/recurring{task name,jobdetails, schedule type {once,cron},cron job? {},start date, end date} return 201 created
patch/task/cancel{task name,task id} return 202 acccepted
post/task/update{task id, ask name,jobdetails, schedule type {once,cron},cron job? {},start date, end date} return202 acccepted
get/tasks/{task id} return 200
get/tasks/ { limit }
get/checkHealth/
get/metrics
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Okay, so the client comes to the API gateway and we talk to a task service. The purpose of task service is to insert a particular request into a database and then the scheduler service takes that particular position. If it's a single time, it will take the single time. If it's a recurring time, it will take that. At any point of time, if you want to do something, the scheduler service will take into picture. It will have an in-memory priority queue along with the write-ahead log so that anything can be handled. Then it will put that into the message queue, which will be a Kafka. And then task executor will try to consume whatever messages are there and it will try to update. And even if multiple retries, the task is not scheduled and it will go to the DLQ. And once the task is completed, it will update the database table that, okay, this task is completed and as well as a notification service has been triggered.
we can keep the API gateway as a routing mechanism, where it will route, authenticate, and also take the rate limiting part, and then it will actually send the one thing to task service. The sole purpose of task service is to put the information in the database, and it will just put all the information like when is the start task single time or recurring time, and all those things. For the recurring time or single time, then we will take the scheduler. The scheduler will scan for the task, and it will take up all the chronicle job, and in that case, all the chronicle jobs will be put into a priority queue, based on the priority of the jobs, it will then send it to the message queue. The message queue will be based on partitions, like you will have partitions, and then there will be consumer groups, like task executor, who will be taking it. The task executor, you can think of it like there will be consumer, which will be running for separate, separate partitions and separate, separate works, and they will be running for it. So this is how they will be executing it. And once they are executed, they are giving down, like they will send a notification.
Interesting approach! How will you ensure that the system can handle bursts of tasks effectively?For a burst of tasks, we will have a, in the tasks in a scheduler service, we can respawn the scheduler with the help of Kubernetes pods, and we can auto do the auto load balancer. And if there are a lot of tasks coming up, we will just respawn more pods and which can take more instances of scheduler, so which can actually take this up into the consideration. And for anything lost, we will actually maintain a write ahead log, and if anything goes on, we will get a new instance and then they will just take the write ahead log as a reference and kind of reprocess it.
One of the approaches we can take is from the Hikari, where we'll keep certain parts up and running, so that even on the spikes, we don't kill all the parts. So suppose like five parts are up and running, and in case any volume come, we can actually go on increasing the parts. We can have, in this way, we will be able to kind of have those tasks, scheduler instance always running, and so this will be a thing where we can, in certain spike, we will be able to at least have some of the scheduler free body.Regarding dead letter queue, the task executor will have each worker inside the task executor can be working on, actually you can say task executor is a consumer group which will consume the tasks from the Kafka message queue and it will retry and if the task is still failed, it will do the DLQ, it will set into a DLQ and which can later down, we can still go for manual review. So the task executor will try for three times with exponential backoff. If then also it fails, it will keep that circuit as break and it will keep on that. Also, we will keep circuit breaker for any task execution. So suppose if certain tasks are failing after a certain time, so we can just say that, okay, we are not passing it out.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
Let's discuss scheduler service. So basically, the task service will be responsible for entering those values in the database. What I mean by this is, there will be tables inside the database where first we will have the jobs, like tasks, which just have the task name, who created it, and all those things, what is the time, and all those things. Second is the recurring job table, where we will keep all the details of like what data, what jobs has to run today. And for one-off jobs, if suppose they want to run it immediately, this direct it will go to the scheduler from the task service. For the recurring jobs, we can get the scheduler to get the data from the database table, and then the scheduler will run the priority queue or ready sorted set, whatever we can call it, and then put tips according to like that in the message queue. And in the message queue, there will be partitions where we will be putting the task ID as the partitions, and then the task executor will have the message consumer group, and then they can do it.
Each Scheduler instance owns a hash range. Task ID is hashed using consistent hashing. Tasks are assigned to Scheduler instances based on hash value. When a Scheduler is added or removed, only tasks in that hash range are reassigned — not the entire keyspace.
On recovery, Scheduler scans DB for tasks where next_run_time < now() and status = pending. Executes the most recent missed run once, recalculates next_run_time from now.
Strategy: Fixed delay — next_run_time = completion_time + interval
Drift detection: If completion_time - scheduled_time > threshold → flag task as drifting → notify via Notification Service
Column needed: next_run_time on tasks table, updated by Task Executor after each run
Duplicate execution prevention ✓ — row level lock
Atomic update ✓ — SELECT FOR UPDATE + UPDATE in same transaction
Multiple scheduler instances ✓ — only one wins the lock
Every task executor will work on the item potency concept. This means you can say the task ID will, the job, task ID will have jobs, so every job execution will be an ID, and if job execution state is pending, it will check in the pending state, otherwise it will check the completed state. So this is how we will be doing it.
To prevent duplicate execution across multiple Scheduler instances, we useSELECT FOR UPDATEon the tasks table. When a Scheduler instance claims a task, it acquires a row-level lock, updates status torunningand setsclaimed_by = scheduler_instance_idin a single transaction. Concurrent instances attempting to claim the same task will block until the lock releases, at which point they see status =runningand skip it.
For recurring tasks we use fixed delay strategy —next_run_time = next_run_at + interval. Drift is detected whencompletion_time - scheduled_time > threshold, triggering an alert via Notification Service. Missed runs are handled by priority — high priority runs all missed executions and alerts, medium runs only the most recent, low priority skips and resumes from now.
Redis Cold Start —
WHERE next_run_at <= now() + 1 minute AND status = 'pending'ZADD NX score=next_run_at member=task_idNX prevents duplicate insertion if multiple Schedulers reload simultaneously. ✓
Recurring table ownership — correct decision:
Task Service = data entry only Scheduler = all computation, next_run_at updates