Loading...
thousands of tasks
minimal latency
high reliability
high availability
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
say 1B active users
1000 req/user/s => 10^6 M QPS
write heavy application, say 90% writes
bandwidth: 0.001 MB / Q X 10^6 M Q/s = 1PB/s
POST /task -d {
taskId
priroity
}
get /task/task_id
PUT /task -d {
taskId
priority
}
delete /task/task_id
client makes a req
it goes through API Gateway where authentication, authorization and rate limiting takes place
task scheduler is central service to view, create, modify or delete a task
task cron scheduler for recurring scheduling tasks
task scheduler sends the request to resource allocator, to allocate task to right queue
to address get requests and fast lookups of task, their scheduling status and their priority , the data would be stored in redis cache
all the tasks will be assigned to queue in a specific order based on the task priority
if a task requested for creation is already their in queue, we check from lookup table instead of putting that again in queue preventing dedup and ensuring idempotency
for edit, we modify the existing entry instead of creating a new one
once a task is processed completely or if there is any failure, client is sent the notification regarding the same
one enhancement that can b done here is adding a dead letter queue, for failed requests and perform retries on them
analytics service can be added to identify which kind of requests are failing more frequently as well as root cause
no sql, redis cache for lookup table for faster reads and massive traffic
queue, like kafka for tasks
scaling:
for massive request handling, queues can be scaled horizontally, keeping tasks to be performed in specific order in same queue
partition keys can be used for task distribution
for high availability, we can do replication on queue and redis in multiple regions for disaster recovery
sharding on queue for faster writes
dead letter queue for analytics and retries
cirucuit breakers can be added to prevent other tasks from getting blocked if certain task is not getting performed
lookup redis table, for idempotency