Requirements

Functional Requirements:

• Allow users to schedule one-time tasks to run at a specified future time.
• Enable users to schedule recurring tasks that repeat on a schedule such as fixed intervals, daily, weekly, monthly, or cron expressions.
• Provide an API that allows users to track, query, list task status, modify schedule or payload, and cancel pending tasks.
• Support sending notifications (email, webhook, Slack) when tasks succeed, fail, or exhaust retries.
• Tasks have priority levels; higher-priority tasks execute before lower-priority ones when resources are contended.

Non-Functional Requirements:

• Assume the task scheduler is internally used. The user base is 100_000, Peak usage 100_000 tasks submitted per day, no peak hours.
• RPS: write 100_000 / 100_000 = 1, Peak: 5, read = 30 RPS, peak 150 RPS
• Latency: avg < 100 ms, P95 < 300 ms.
• HA: 99.99%, for create, schedule, read and notifications.
• Storage: retention 10yrs, 100_000 * 1kb * 400 * 10 = 400 GB
• P0: The system should be scalable to adapt to high demand
• P0: The system should schedule the task on time (+- 10s diff).
• P1: The system should be able to handle different workloads.
• P1: Notifications should be sent within 10 minutes, max 1 hour.

API Design

The system is web-based, so all APIs should be authenticated against JWT.

POST /task

req: {

dedup_id,

schedule: (cron format, task is one time if left empty),

delay: (11 timestamp - how long to wait until running),

payload: (an executable, multipart binary)

priority

}

resp: {

task_id

}

PUT /task

req: {

task_id,

schedule: (cron format, task is one time if left empty),

delay: (11 timestamp - how long to wait until running),

payload: (an executable, multipart binary)

priority

}

PUT /task/status

req: {

status: enable/disable

}

DELETE /task?id=...

GET /task?id=...

resp: one task detail, and running status, schedule history

{

task_id

detail: {...}

status: (created/to be scheduled/running/finished),

history: [ {

scheduled_timestamp, status (success/failed)

} ]

}

GET /tasks

resp: task list

High-Level Design

Data model:

UserTask

id(pk), user_id(fk), dedup_key(uk), payoad_s3_path, delay, priority, cron, status, task_state, created_time, updated_time

SchedulerTask

task_id, execution_id, user_id, timestamp, state

User(id, name, etc.)

State Transition:

created -> waiting/waiting_for_resource -> running -> success/failed/no_resource

On a high-level the system takes a task and stores it in the database, and sends a message to a scheduler.

A backend scheduler sorts all the enabled tasks based on priority in memory and creates scheduler tasks, executes them, then update the status based on execution result, send them back to the MQ to notify the Task service.

Detailed Component Design

Task Service:

• Takes the tasks create/update/delete requests and updates the database, sends messages to the scheduler to notify it.
• Subscribe to the execution result, update task status and state, and send notifications.

Scheduler:

• Sort the enabled tasks in the memory based on priority for each user (each user gets their own list). and creates
• For each task, the scheduler will create an execution event and send it to MQ for the executor to sub and exec. If the executor fails to execute the task, it will do an exponential retry for 5 times, and if it keeps failing, it will update the scheduler task state and send the result to the MQ to notify the Task Service.
• For a one-time task, the scheduler will create a single execution event and remove the task from the list.
• The Scheduler will inspect the resource in the K8s cluster, and only execute the top task on the list if there is not enough resource for all tasks. Since the tasks for each user is ordered by priority in-memory, this process is fast.
• If the resource is not enough and system cannot satisfy the SLA (+- 10s), the executor will update the task status to waiting_for_resource state.

HA Design:

1. The Task Service and Scheduler should have at least 3 instances, even though the RPS is low. The Scheduler is stateful, but the in-memory state can be rebuilt from the database once spin up.
2. The database should have at least one follower node
3. The MQ should have at least one replica

Scalability:

One critical point is the task resource on k8s. We can do the following:

1. Try to clean up zombie tasks that use ZERO CPU and ZERO network for a very long time - notify the user to ask them to delete it.
2. Scale out the cluster horizontally
3. Enable CPU share in the cluster.