System requirements

Functional:

• Need to be able to schedule ad-hoc tasks for a specific time
• Need to be able to schedule cron job

Non-Functional:

• Minimal delay from scheduled time to start of execution
• Reliable - retry jobs that have infra error

Capacity estimation

Can execute thousands of tasks

API design

PUT /scheduleAdHoc

{

time: time

repo: string

file: something.sh

}

PUT /scheduleCron

{

cron: string

repo: string

file: something.sh

}

Database design

We will have a key-value store that maps timestamp to job info. In a separate collection we will map cron string to job info.

High-level design

We will have a replicated service for scheduling jobs. We will also have a cronjob reader service that scans all the cronjob strings periodically and adds entries to the timestamp=>job collection. We will maintain a pool of worker services that are ready to execute jobs immediately. We will have a job manager service that, every second, checks the DB to see what jobs, if any, need to be executed, and gives the job to an available worker.

Request flows

Client schedules a task ad-hoc. The scheduler writes the job to the DB. Job manager service periodically reads from the DB to see what jobs need to be run and assigns work to workers. Once the worker is done the manager clears the job from the DB.

Recurring task scheduling is similar, except the cronjob reader service needs to repeatedly schedule the individual task executions in the database.

Detailed component design

If the job scheduling service receives a recurring request, it schedules 1 hour's worth of requests in case the cronjob reader service takes some time to notice the new cronjob.

Cronjob reader service periodically scans the list of cronjobs and schedules an hour's worth of requests, if they haven't been scheduled already.

Job manager needs to keep track of worker health and re-execute a job on a new worker if a worker goes down.

Trade offs/Tech choices

Jobs may not execute at the exact second they are scheduled for. The key-value store is scalable and simple but it may be inconvenient to not be able to query jobs that are scheduled within a time window.

Failure scenarios/bottlenecks

Jobs may get backed up if there are not enough workers in the pool that are idle, as it takes some time to start up workers. Also, if a worker goes down while executing a task, the actions that the task did may be repeated if the task is retried.

Future improvements

Need to implement deletion of recurring task.