Loading...
Schedule Tasks
Delete Tasks
Scalability
Availability
Reliability
Fault Tolerant
Low latency/Performance
Data centre level task scheduler which has multiple clients and it executes tasks and assigns resources for the client according to what a client has paid for. We expect our task scheduler to receive billions of tasks which it should be able to handle graciously.
addTask(taskId, List<Resource>)
deleteTask(taskId)
Relational Database like MySQL for task metadata and client metadata. This allows structured data storage and ACID properties.
Key-Value Store (NoSQL) database to store taskId and the task itself. NoSQL DB allows unstructured data storage and fast retrieval.
Components :
Load Balancer
Rate Limiter
Task submitter
Task Manager
Application Servers/Task Executors
Sequencer
Relational Database
Distributed Messaging queue (There will be mainly 3 types, one for urgent tasks, one for tasks that are dependant on other tasks and one for independent tasks.)
Monitoring
Administrator
Client
We have clients, which will submit tasks or delete tasks. Rate limiter pass or reject requests based on client's permitted request threshold. Load Balancers will distribute the requests appropriately among the task submitters. Task submitters will have business logic and validations around the request. Task Submitter will call the sequencer to generate a unique ID for each task and add the task to the message queue and will store the task and client metadata in MySQL. After this, the Application Servers will fetch tasks from the distributed queues depending on various factors like task urgency and dependencies. It will allocate resources to the tasks based on the resources required for the task and execute the task. The monitoring server keeps track of health of all the resources and informs the administrator if any resources are down and needs to be replaced.
Client submits a task along with details about the resources required by the task, urgency or dependency of the tasks. Then rate limiter checks the threshold for the client and send the request to the load balancer. Load balancer then send the request to any available task submitter. Task submitter will generate a unique Id for the task by calling the sequencer, and store the task and client metadata in MySQL. Task Manager will check the details of the task and add it in the appropriate queue (urgent, dependent, independent task queues). Then Application servers/task executors will fetch tasks from the these queues depending on urgency and dependencies of the tasks. It will allocate resources to each tasks, and if the task is completed then it sends the success status to the task manager which then deletes the task from the queue. But if the task is failed, then task manager will add the task again in the queue for the retry number allowed for the task. And if the tasks takes longer to execute than specified time or needs more resources than what the client had specified, task will be stopped and and error will be sent to the client. The monitoring service continuously tracks the health of resources and informs administrator if anything needs attention and also collects data for monitoring purposes.
If a client submits a delete request then task manager will delete the task from the queue.
With billions of tasks being submitted and processed by the task scheduler, we need robust storage arrangements. So single servers of databases will not suffice and we will also need to shard our data. We can opt for horizontal sharding of data based on task Id ranges. And replicate data with the replication factor of 3 where one server will be the leader and other two will be the followers. One server will entertain the write requests and other two can serve the read requests. And we will have an asynchronous replication among leader and follower shards and servers of the databases.
A sequencer is being used for generating unique Ids for the task. But single server will increase the latency but multiple servers might mess up the uniqueId generation. In this case, we have used distributed sequencer which will have range of uniqueIds allocated to each server. This will greatly improve latency in the design.
We have chosen MySQL database for storing client and task metadata, while relational databases are not typically known to be very fast, but for the metadata storage and ACID properties we need a structured relational DB.
We chose NoSQL DB for storing tasks as tasks will be added by different clients and might not have a uniform structure. So we chose NoSQL database which will also be fast
Task Idempotency is something that we need to account for. Say for example, in case of banking services, tasks absolutely need to be idempotent or it can lead to severe issues. Like if a transaction was successful but the acknowledgment failed, and if the task scheduler retries such a task and the transaction happened twice instead of once, this will be a huge error for the banking application.
We can make use of an idempotency key, which will basically be a uniqueId which will store in a separate database(NoSQL database). So even when the acknowledgment fails, when the task is retried, we can first check if idempotency key of the task is present in the database, if it is then we wont execute the task again. If its not there then we know that the transaction had really failed last time and we can go ahead with executing the task again.