System requirements


Functional:

  • users can create tasks which can be executed in certain time
  • users can read/update/delete those tasks
  • users should be able to see the task execution status
  • tasks can be cron jobs



Non-Functional:

  • reliable: the tasks can be re-executed if failed
  • Consistency: Job results should be consistent, ensuring that jobs are executed once (or with minimal duplication).
  • scalability: the system should be easily scaled up with more usage


Capacity estimation

Assume we have ADU 1M and 10 write per day and 100 read per day

  • 100 write qps and 1000 read qps, for peak, it could be 1k write qps and 10k read qps


Assume each tasks should be less than 1MB, we can have 3.6 PB data each year




API design


  • createTask(user_id, scheduled_time, task_content) -> task_id
  • getTask(task_id) -> status, result
  • updateTask(task_id, sceduled_time, task_content) -> status
  • deleteTask(task_id) -> status


Database design


task(task_id, user_id, create_time, scheduled_time, recurring, content) # it can be recurring


execution(execution_id, task_id, execution_status, failed_time, result)

# execution_status can be: new, in-progress, success, non-retriable_error, retriable_error


task 1 ... n execution


High-level design

The whoe system contains several parts:

  • Load balancer and API Gateway is responsible for rate limiting, authn, authz, xxxx etc
  • Task management service is responsible to CRUD the tasks and write that into Database
  • The message queue(Kalfka) is help balance load of scheduled execution
  • The task scheduler server is responsible to create executions scheduled at the current time point
  • The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database





Request flows

The whoe system contains several parts:

  • the task scheduler service has multiple workers to poll DB partitions and
    • find some tasks need to be run
    • create a new execution in the table
    • push the execution into the message queue
  • The task executor server is responsible to pull the execution from the queue and assign to its worker. When it is done, update the result into the database. If it is failed, push that back to the message queue


Detailed component design

ā‰ˆ

  • Inside task executor service and task scheduler service, they have multiple workers running(to poll DB or to run tasks), we can have cooridination service to monitor their health by hearbeat
  • Scheduling algorithm, round-robin, priority-based scheduling(based on the users' priority or the task execution duration) using priority queue
  • Failure retry, the failed execution will be pushed back the message queue and we can implement an exponential back-off algorithm for the next retry
  • Timezone: we will unify all the times to be UTC based


Trade offs/Tech choices





Failure scenarios/bottlenecks

Any task execution can failed in the task executor worker for any reasons: out of power, network outage, out of CPU/memory resource. The task executor service will watch each execution and add it back the message queue if it is failed but still retriable.


The bottlenecks can be:

  • DB: we need more relational DB if we have more traffic
  • we probably need more machines for executions





Future improvements

With higher traffic, we need to

  • sharding our database and in the meanwhile,
  • we need to scale up the workers in task scheduler service
  • we need to scale up the workers in task executor service


Add monitoring and alerting on the executoin failure


The execution result can be stored a nosql or even blob store and then store a link in db


We can add a redis cache for readTask API