System requirements

Functional:

• users can create tasks which can be executed in certain time
• users can read/update/delete those tasks
• users should be able to see the task execution status
• tasks can be cron jobs

Non-Functional:

• reliable: the tasks can be re-executed if failed
• Consistency: Job results should be consistent, ensuring that jobs are executed once (or with minimal duplication).
• scalability: the system should be easily scaled up with more usage

Capacity estimation

Assume we have ADU 1M and 10 write per day and 100 read per day

• 100 write qps and 1000 read qps, for peak, it could be 1k write qps and 10k read qps

Assume each tasks should be less than 1MB, we can have 3.6 PB data each year

API design

• createTask(user_id, scheduled_time, task_content) -> task_id
• getTask(task_id) -> status, result
• updateTask(task_id, sceduled_time, task_content) -> status
• deleteTask(task_id) -> status

Database design

task(task_id, user_id, create_time, scheduled_time, recurring, content) # it can be recurring

execution(execution_id, task_id, execution_status, failed_time, result)

# execution_status can be: new, in-progress, success, non-retriable_error, retriable_error

task 1 ... n execution

High-level design

The whoe system contains several parts:

• Load balancer and API Gateway is responsible for rate limiting, authn, authz, xxxx etc
• Task management service is responsible to CRUD the tasks and write that into Database
• The message queue(Kalfka) is help balance load of scheduled execution
• The task scheduler server is responsible to create executions scheduled at the current time point
• The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database

Request flows

The whoe system contains several parts:

• the task scheduler service has multiple workers to poll DB partitions and
  • find some tasks need to be run
  • create a new execution in the table
  • push the execution into the message queue
• The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database. If it is failed, push that back to the message queue

Detailed component design

• Inside task executor service and task scheduler service, they have multiple workers running(to poll DB or to run tasks), we can have cooridination service to monitor their health by hearbeat
• Scheduling algorithm, round-robin, priority-based scheduling(based on the users' priority or the task execution duration) using priority queue
• Failure retry, the failed execution will be pushed back the message queue and we can implement an exponential back-off algorithm for the next retry
• Timezone: we will unify all the times to be UTC based

Trade offs/Tech choices

• DB choice: we will use relationable DB as we want the task to be executed only once.

Failure scenarios/bottlenecks

Any task execution can failed in the task executor worker for any reasons: out of power, network outage, out of CPU/memory resource. The task executor service will watch each execution and add it back the message queue if it is failed but still retriable.

The bottlenecks can be:

• DB: we need more relational DB if we have more traffic
• we probably need more machines for executions

Future improvements

With higher traffic, we need to

• sharding our database and in the meanwhile,
• we need to scale up the workers in task scheduler service
• we need to scale up the workers in task executor service

Add monitoring and alerting on the executoin failure

The execution result can be stored a nosql or even blob store and then store a link in db

We can add a redis cache for readTask API