Loading...
thousands of tasks
minimal latency
high reliability
high availability
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
say 1B active users
task execution
1 req/user/day => 10^9/24x3600 = 10^9/8.64x10^4 = 10^4 QPS approx
task scheduling
1 req/user/day => 10^9/24x3600 = 10^9/8.64x10^4 = 10^4 QPS approx
write heavy application, say 90% writes
bandwidth: 0.001 MB / Q X2x 10^4 Q/s = 20MB/s = 2 TB/day approx
POST /task?schedule_recurring=true -d {
schedule
}
response: {
task_id
schedule
task_duration
task_duration
}
POST /task?schedule_recurring=false -d {
schedule
}
response: {
task_id
schedule
task_duration
}
get /task/task_id
response: {
task_id
schedule
task_duration
}
PUT /task -d {
taskId
schedule
task_duration
}
response: {
task_id
schedule
task_duration
}
delete /task/task_id
client makes a req
it goes through API Gateway where authentication, authorization and rate limiting takes place
task scheduler is central service to view, create, modify, delete or schedule a task
to address get requests and fast lookups of task, their scheduling status and their priority , the data would be stored in redis cache
all the tasks will be assigned to queue in a specific order based on the task priority.
task priority is decided based on the schedule, lesser the run_at time, higher the priority
for tie breaker, for the task execution req at same schedule, priority would be based on task_duration basis, lesser the task duration, higher the priority to avoid smaller tasks for waiting for longer duration
for again the ties breaker here, it would be first come first serve basis
for concurrent requests, locks would be applied to decide on priority and maintain right priority. counter.
if a task requested for creation is already their in queue, we check from lookup table instead of putting that again in queue preventing dedup and ensuring idempotency
for edit, we modify the existing entry instead of creating a new one
task executor: is responsible for picking up highest priority tasks from the queue and its task execution and returning status of task, and update its status to running in db
once a task is added to the queue, its lifecyle is controller by the task controller , and updates the task execution status in db as well as perform retries on failed tasks, removes completed tasks from the queue
it will keep polling task executor, to get the status of task execution and update the statedb on status change
once a task is processed completely or if there is any failure, it sends the the notification regarding the same thorugh notification system
we can add dead letter queue, for failed requests and perform retries on them
analytics service can be added to identify which kind of requests are failing more frequently as well as root cause
redis cache for lookup table for faster reads and massive traffic for recently accessed data
postgresql db for storing task information and execution status
(postgres-db to ensure atomic transactions ans easy joins on tables)
schema:
task_details:
task_id: int
schedule: data_time
execution_status: [RUNNING, COMPLETE, FAILED, STUCK]
recurring: bool
task_duration
recurring_task_details:
task_id
run_at
next_run_at
queue, like kafka for maintaing tasks in poper priority order
sequeuence:
clientReq->task-scheduler->task-controller->task-executor->task-controller->notfication service and anaytics service
task scheduler: client req directed to task-scheduler
task scheduler inserts entry in postgres-db, frequently accessed data is added to redis cache
task controller runs every 5 min to check for upcoming tasks and inserts them to the queue on the basis of priority(high priority= lower-scheduled-time->lower-task-duration->FIFO)
task executor, picks up high priority tasks, update its status to RUNNING in db
task controller polls for running tasks from task executor, and updates the execution status in db accordingly as well as updates the queue
scaling:
for massive request handling, queues can be scaled horizontally, keeping tasks to be performed in specific order in same queue
task controller: can be scaled horizontally, but for edit requests, we need to avoid write conflicts:
so we can keep one writer server and other read replicas, to avoid write conflicts
for task controller, it can be scaled horizontally seamlessly, due to kafka handling the list of queues
now each replica can pull tasks from specific partition, avoiding any conflicts on picking up same task
each time a task is modified, and it is inserted again n queue, their maybe invalid older entries in queue as well, to avoid their execution, we validate the entries from postgres-db to check if it is the last modified version of the task, and only then pick it up
partition keys can be used for task distribution
for high availability, we can do replication on queue and redis in multiple regions for disaster recovery
sharding on queue for faster writes
dead letter queue for analytics and retries
cirucuit breakers can be added to prevent other tasks from getting blocked if certain task is not getting performed
lookup redis table, for idempotency
if redis restarts, it starts updating cache from then
for missing recurring runs say due to task scheduler crash, since we manitain run_at, it would help task scheduler pick up that task in next time when task scheduler recovers, avoiding any miss there