Requirements

Functional Requirements:

• Allow users to schedule one-time tasks to run at a specified future time.
• Enable users to schedule recurring tasks that repeat on a schedule such as fixed intervals, daily, weekly, monthly, or cron expressions.
• Provide an API that allows users to track, query, list task status, modify schedule or payload, and cancel pending tasks.
• Support sending notifications (email, webhook, Slack) when tasks succeed, fail, or exhaust retries.
• Tasks have priority levels; higher-priority tasks execute before lower-priority ones when resources are contended.

Non-Functional Requirements:

• Reliability, so jobs are executed the exact number of times and notifications are delivered
• Scalability, so the system can absorb more users without breaking the system

API Design

It is important to highlight the different status of the job before explaining the endpoints:

1. Created: When the job is created, either by API post or by the worker if the job is recurrent.
2. Scheduled: When the schedulers have found a created job, mark it as scheduled.
3. Running: When the workers have found a scheduled job and the time of the execution has arrived.

It would have the following endpoints:

POST /job: Creates a job in the database storing the interval and setting a priority (high, medium, low). Returns an ID for future actions by users and 201 if successful.

GET /job/{id}: Returns a job by ID, 200 if successful 404 if not found.

PUT /job/{id}: Amends an existing job. This might include cancelling a execution for a recurring job as far as the job is not in a status "running" (409 in this case). Returns 204.

DELETE /job/{id}: Deletes a job, returning 204. Returns an error 409 if the status of the job is "running". Removes the job from the database.

High-Level Design

A user calls the POST endpoint to create a job, and it is created in the database.

A group of schedulers do polling on the database looking for jobs in status "created" and time greater than now. When there is less than 2 minutes to run the job changes the status to "scheduled" in the db and sends a message to a queue depending on the priority of the job.

A group of workers consume from the priority queues and wait for the time to execute the job, that should not exceed 2 minutes. We use idempotency keys at this level to identify repeated jobs. We run the job, in case of errors we implement exponential backoff with jitter so we do not retry at the same time for every failed job. After that, we notify the user with either succeeded/failed job.

In the case of email, since we need a external provider, we use a dedicated queue so emails can be sent using that.

The reviewers are there to identify possible inconsistencies in the system. For instance, if a job has been running for more of 5 minutes, a reviewer will unblock the job and put it back to the "created" status with the defined cadence (for instance, if the job runs every day, the created_at field in the db will be updated to the next day).

Detailed Component Design

With regards to the database, we use SQL since we need ACID in the transactions to guarantee consistency since there are several elements (schedulers, workers and reviewers) updating the same rows of the database.

When we create a job, we set a column created_at that is used to identify the cadence. This value will be changed depending on the intervals also defined when creating the job.

We guarantee priority by defining 3 queues. If resources are contended the workers will only execute jobs in the high priority queue.

We define a execution limit per job of 5 minutes. The reviewers are there to amend the status of the job depending on this.

Cancelling-wise there are 2 levels. Deleting the job (it wont run anymore) or cancelling the next execution (via PUT endpoint).

We scale the different components (API, workers, schedulers reviewers) depending on the load of the system. Database-wise we can use sharding to improve writes and also replicas to improve performance for the GET endpoints.