Requirements

Functional Requirements:

• Allow users to schedule one-time tasks to run at a specified future time.
• Enable users to schedule recurring tasks that repeat on a schedule such as fixed intervals, daily, weekly, monthly, or cron expressions.
• Provide an API that allows users to track, query, list task status, modify schedule or payload, and cancel pending tasks.
• Support sending notifications (email, webhook, Slack) when tasks succeed, fail, or exhaust retries.
• Tasks have priority levels; higher-priority tasks execute before lower-priority ones when resources are contended.

Non-Functional Requirements:

• Assume the task scheduler is internally used. The user base is 100_000, Peak usage 100_000 tasks submitted per day, no peak hours.
• RPS: write 100_000 / 100_000 = 1, Peak: 5, read = 30 RPS, peak 150 RPS
• Latency: avg < 100 ms, P95 < 300 ms.
• HA: 99.99%, for create, schedule, read and notifications.
• Storage: retention 10yrs, 100_000 * 1kb * 400 * 10 = 400 GB
• P0: The system should be scalable to adapt to high demand
• P0: The system should schedule the task on time (+- 10s diff).
• P1: The system should be able to handle different workloads.
• P1: Notifications should be sent within 10 minutes, max 1 hour.

API Design

The system is web-based, so all APIs should be authenticated against JWT.

POST /task

req: {

dedup_id,

schedule: (cron format, task is one time if left empty),

delay: (11 timestamp - how long to wait until running),

docker_image_name

docker_image_tag

priority

}

resp: {

task_id

}

PUT /task

req: {

task_id,

schedule: (cron format, task is one time if left empty),

delay: (11 timestamp - how long to wait until running),

docker_image_name

docker_image_tag

priority

}

PUT /task/status

req: {

status: enable/disable

}

DELETE /task?id=...

GET /task?id=...

resp: one task detail, and running status, schedule history

{

task_id

detail: {...}

status: (created/to be scheduled/running/finished),

history: [ {

scheduled_timestamp, status (success/failed)

} ]

}

GET /tasks

resp: task list

High-Level Design

Data model:

UserTask

id(pk), user_id(fk), dedup_key(uk), docker_image_name, docker_image_tag, delay, priority, cron, status, task_state, created_time, updated_time

SchedulerTask

task_id, execution_id, user_id, timestamp, state

User(id, name, etc.)

State Transition:

created -> waiting/waiting_for_resource -> running -> success/failed/no_resource

On a high-level the system takes a task and stores it in the database, and sends a message to a scheduler.

A backend scheduler sorts all the enabled tasks based on priority in Redis and creates scheduler tasks, executes them, then updates the status based on execution result, send them back to the MQ to notify the Task service.

Detailed Component Design

Task Service:

• Takes the tasks create/update/delete requests and updates the database, sends messages to the scheduler to notify it.
• Subscribe to the execution result, update task status and state, and send notifications.

Scheduler:

• Sort the enabled tasks in the Redis based on priority for each user (each user gets their own list). and creates
• For each task, the scheduler will create an execution event and send it to MQ for the executor to sub and exec. If the executor fails to execute the task, it will do an exponential retry for 5 times, and if it keeps failing, it will update the scheduler task state and send the result to the MQ to notify the Task Service.
• For a one-time task, the scheduler will create a single execution event and remove the task from the list.
• The Scheduler will inspect the resource in the K8s cluster, and only execute the top task on the list if there is not enough resource for all tasks.
• If the resource is not enough and system cannot satisfy the SLA (+- 10s), the executor will update the task status to waiting_for_resource state.

Recurring Task Handling:

In the Redis task list, it is the user task. For each recurring user task, the scheduler would monitor its time, when it matches the current time, create a scheduler task for it with an execution_id, and send it to the MQ for the executor to subscribe.

Dedup:

The dedup key is generated by the web-end, since the dedup field is a uk in the database, the Task Service would be able to prevent double-creating the task.

HA Design:

1. The Task Service and Scheduler should have at least 3 instances, even though the RPS is low.
2. The database should have at least one follower node
3. The MQ should have at least one replica

Scalability:

One critical point is the task resource on k8s. We can do the following:

1. Try to clean up zombie tasks that use ZERO CPU and ZERO network for a very long time - notify the user to ask them to delete it.
2. Scale out the cluster horizontally
3. Enable CPU share in the cluster.

Another is the scheduler partitioning:

1. partition the task message by id in the MQ, put all schedulers in one consumer group for the topic, let them coordinate the subscription. Same for the executor - the scheduler publishes scheduling tasks and partitions by execution_id, and the executor consumers consume in one group.