Requirements

Functional Requirements:

• Allow users to schedule one-time tasks to run at a specified future time.
• Enable users to schedule recurring tasks that repeat on a schedule such as fixed intervals, daily, weekly, monthly, or cron expressions.
• Provide an API that allows users to track, query, list task status, modify schedule or payload, and cancel pending tasks.
• Support sending notifications (email, webhook, Slack) when tasks succeed, fail, or exhaust retries.
• Tasks have priority levels; higher-priority tasks execute before lower-priority ones when resources are contended.

Non-Functional Requirements:

• Reliability, so jobs are executed the exact number of times and notifications are delivered
• Scalability, so the system can absorb more users without breaking the system

API Design

It is important to highlight the different status of the job before explaining the endpoints:

1. Created: When the job is created, either by API post or by the worker if the job is recurrent.
2. Scheduled: When the schedulers have found a created job, mark it as scheduled.
3. Running: When the workers have found a scheduled job and the time of the execution has arrived.

It would have the following endpoints:

POST /job: Creates a job in the database storing the interval and setting a priority (high, medium, low). Returns an ID for future actions by users and 201 if successful.

GET /job/{id}: Returns a job by ID, 200 if successful 404 if not found.

PUT /job/{id}: Amends an existing job. This might include cancelling a execution for a recurring job as far as the job is not in a status "running" (409 in this case). Returns 204.

DELETE /job/{id}: Deletes a job, returning 204. Returns an error 409 if the status of the job is "running". Removes the job from the database.

High-Level Design

A user calls the POST endpoint to create a job, and it is created in the database.

A group of schedulers do polling on the database looking for jobs in status "created" and time less or equal than now. When the criteria is met the scheduler changes the status to "scheduled" in the db and sends a message to a queue depending on the priority of the job. The row is blocked while the status update is happening so other schedulers skip it.

A group of workers consume from the priority queues. We use idempotency keys at this level to identify repeated jobs. We run the job, in case of errors we implement exponential backoff with jitter so we do not retry at the same time for every failed job. After that, we notify the user with either succeeded/failed job.

In the case of email, since we need a external provider, we use a dedicated queue so emails can be sent using that.

The reviewers are there to identify possible inconsistencies in the system. For instance, if a job has been running for more than the configured timeout, a reviewer will unblock the job and put it back to the "created" status with the defined cadence (for instance, if the job runs every day, the execute_at field in the db will be updated to the next day).

Detailed Component Design

With regards to the database, we use SQL since we need ACID in the transactions to guarantee consistency since there are several elements (schedulers, workers and reviewers) updating the same rows of the database. We will have an index on (status, execute_at) since it is used by several components.

When we create a job, we set a column created_at and execute_at that is used to identify the cadence. execute_at value will be changed depending on the intervals also defined when creating the job. execute_at will have a small jitter to avoid running to many jobs at the same time.

We guarantee priority by defining 3 queues. If resources are contended the workers will only execute jobs in the high priority queue.

We define a configurable execution limit per job. The reviewers are there to amend the status of the job depending on this.

Cancelling-wise there are 2 levels. Deleting the job (it wont run anymore) or cancelling the next execution (via PUT endpoint).

We scale the different components (API, workers, schedulers reviewers) depending on the load of the system. Database-wise we can use sharding to improve writes and also replicas to improve performance for the GET endpoints.

HA, reliability, scale and growth: Redundancy and failover is covered by the database replicas. With regards to other components:

1. If a scheduler fails doing its job, another scheduler will pick up the task and update the job accordingly.
2. If a worker goes down when running a task, the reviewer will detect the inconsistency in the database and will update the job accordingly so it can be picked up by another worker (configured timeout)
3. We will use DLQs for failed jobs (those were exponential backoff was exhausted) for the reviewers to recreate in the jobs db.
4. A known limitation is that if a worker executes and goes down before updating the db the job can be executed more than once, so our system guarantees at-least-once. This can be mitigated if the external system (email, slack) also has idempotency keys.

We use a load balancer to route requests to API servers, that will auto-scale depending on the usage (horizontal scaling)

Recurring tasks: Rely on 2 columns from the database, execute_at (next execution) and interval. The scheduler publishes in the priority queues only old jobs or jobs that are supposed to happen now. The the workers consume the job and execute it, updating the job status to "running". Once it is done, the worker updates the row to the "created" status and "execute_at" using the interval.