Requirements

Functional Requirements:

• Allow users to schedule one-time tasks to run at a specified future time.
• Enable users to schedule recurring tasks that repeat on a schedule such as fixed intervals, daily, weekly, monthly, or cron expressions.
• Provide an API that allows users to track, query, list task status, modify schedule or payload, and cancel pending tasks.
• Support sending notifications (email, webhook, Slack) when tasks succeed, fail, or exhaust retries.
• Tasks have priority levels; higher-priority tasks execute before lower-priority ones when resources are contended.

Non-Functional Requirements:

• Reliability, so tasks are executed the exact number of times and notifications are delivered
• Scalability, so the system can absorb more users without breaking the system

API Design

It is important to highlight the different status of the task before explaining the endpoints:

1. Created: When the task is created, either by API post or by the worker if the task is recurrent.
2. Scheduled: When the schedulers have found a created task, mark it as scheduled.
3. Running: When the workers have found a scheduled task and the time of the execution has arrived.
4. Succeeded/failed: If the worker suceeds/fails executing the task.

It would have the following endpoints:

POST /task: Creates a task in the database storing when the first execution should happen, the type (one-off, time interval, cron job, monthly, daily) and a priority (high, medium, low). Returns an ID for future actions by users and 201 if successful. By default, records for 6 months will be created (so if type monthly, 6 rows will be added to the database using the execute_at field with a jitter).

GET /task/{id}: Returns a task by ID, 200 if successful 404 if not found.

PUT /task/{id}: Amends an existing task. This might include cancelling a execution for a recurring task as far as the task is not in a status "running" (409 in this case). Returns 204.

DELETE /task/{id}: Deletes a task, returning 204. Returns an error 409 if the status of the task is "running". Removes the task from the database.

GET /health: Returns 200 if the API is healthy, 503 if not breaking down the status by component (workers, schedulers, reviewers, DB)

GET /metrics: Metrics of the system expose for observability. Relies on a time series db (like influxdb) that I wont detail here since it is not part of the core functionality.

High-Level Design

A user calls the POST endpoint to create a task, and it is created in the database.

A group of schedulers do polling on the database looking for tasks in status "created" and time less or equal than now. When the criteria is met the scheduler changes the status to "scheduled" in the db and sends a message to a queue depending on the priority of the task. The row is blocked while the status update is happening so other schedulers skip it.

A group of workers consume from the priority queues. We use idempotency keys at this level to identify repeated tasks. We run the task, in case of errors we implement exponential backoff with jitter so we do not retry at the same time for every failed task. After that, we notify the user with either succeeded/failed task.

In the case of email, since we need a external provider, we use a dedicated queue so emails can be sent using that.

The reviewers are there to identify possible inconsistencies in the system. For instance, if a task has been running for more than the configured timeout, a reviewer will unblock the task and put it back to the "created" status with the defined cadence (for instance, if the task runs every day, the execute_at field in the db will be updated to the next day).

On top of the API, the other component will expose healthcheck endpoints to be orchestrated by the API servers that expose the healthcheck endpoint for the user.

Detailed Component Design

With regards to the database, we use SQL since we need ACID in the transactions to guarantee consistency since there are several elements (schedulers, workers and reviewers) updating the same rows of the database. We will have an index on (status, execute_at) since it is used by several components.

When we create a task, we set a column created_at and execute_at that is used to identify the cadence. execute_at value will be changed depending on the intervals also defined when creating the task. execute_at will have a small jitter to avoid running to many tasks at the same time.

We guarantee priority by defining 3 queues. If resources are contended the workers will only execute tasks in the high priority queue.

We define a configurable execution limit per task. The reviewers are there to amend the status of the task depending on this.

Cancelling-wise there are 2 levels. Deleting the task (it wont run anymore) or cancelling the next execution (via PUT endpoint).

We scale the different components (API, workers, schedulers reviewers) depending on the load of the system. Database-wise we can use sharding to improve writes and also replicas to improve performance for the GET endpoints.

HA, reliability, scale and growth: Redundancy and failover is covered by the database replicas. With regards to other components:

1. If a scheduler fails doing its task, another scheduler will pick up the task and update the task accordingly.
2. If a worker goes down when running a task, the reviewer will detect the inconsistency in the database and will update the task accordingly so it can be picked up by another worker (configured timeout)
3. We will use DLQs for failed tasks (those were exponential backoff was exhausted) for the reviewers to recreate in the tasks db.
4. A known limitation is that if a worker executes and goes down before updating the db the task can be executed more than once, so our system guarantees at-least-once. This can be mitigated if the external system (email, slack) also has idempotency keys.

We use a load balancer to route requests to API servers, that will auto-scale depending on the usage (horizontal scaling)

Recurring tasks: Rely on 2 columns from the database, execute_at (next execution) and interval. The scheduler publishes in the priority queues only old tasks or tasks that are supposed to happen now. The the workers consume the task and execute it, updating the task status to "running". Once it is done, the worker updates the row to the "created" status and "execute_at" using the interval.

Missed recurring runs will be addressed by the schedulers since their polling checks by execute_at that are either now or older. Then the flow will work as usual.

Midnight spike: Already explained, a jitter is added to every execution in the execute_at attribute so they do not run at the exact same time.

Idempotency: Already explained above. Applied to the workers. If a worker consumes from the queue a message with an idempotency key already seen (checking this against the database, suceeded/failed status) discards the messsage. If it is the first time it is seen or the status is different to the above, retries the execution.