System requirements

Functional:

• users can create tasks which can be executed in certain time
• users can read/update/delete those tasks
• users should be able to see the task execution status
• tasks can be cron jobs

Non-Functional:

• reliable: the tasks can be re-executed if failed
• Consistency: Job results should be consistent, ensuring that jobs are executed once (or with minimal duplication).

Capacity estimation

Assume we have ADU 1M and 10 write per day and 100 read per day

• 100 write qps and 1000 read qps, for peak, it could be 1k write qps and 10k read qps

Assume each tasks should be less than 1MB, we can have 3.6 PB data each year

API design

• createTask(user_id, scheduled_time, task_content) -> task_id
• getTask(task_id) -> status, result
• updateTask(task_id, sceduled_time, task_content) -> status
• deleteTask(task_id) -> status

Database design

task(task_id, user_id, create_time, scheduled_time, recurring, content) # it can be recurring

execution(execution_id, task_id, execution_status, failed_time, result)

# execution_status can be: new, in-progress, success, non-retriable_error, retriable_error

task 1 ... n execution

High-level design

The whoe system contains several parts:

• Load balancer and API Gateway is responsible for rate limiting, authn, authz, xxxx etc
• Task management service is responsible to CRUD the tasks and write that into Database
• The message queue(Kalfka) is help balance load of scheduled execution
• The task scheduler server is responsible to create executions scheduled at the current time point
• The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database

Request flows

The whoe system contains several parts:

• the task scheduler service has multiple workers to poll DB partitions and
  • find some tasks need to be run
  • create a new execution in the table
  • push the execution into the message queue
• The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database. If it is failed, push that back to the message queue

Detailed component design

• DB choice: we will use relationable DB as we want the task to be executed only once.
• Inside task executor service and task scheduler service, they have multiple workers running(to poll DB or to run tasks), we can have cooridination service to monitor their health by hearbeat
• Scheduling algorithm, round-robin, priority-based scheduling(based on the users' priority or the task execution duration) using priority queue
• Exponential retry, we can do exponential retry on the failed execution

Trade offs/Tech choices

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

With higher traffic, we need to

• sharding our database and in the meanwhile,
• we need to scale up the workers in task scheduler service
• we need to scale up the workers in task executor service

Add monitoring and alerting on the executoin failure

The execution result can be stored a nosql or even blob store and then store a link in db

We can add a redis cache for readTask API