Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
We require a system that will make the next things:
List the key non-functional requirements (performance, scalability, reliability, etc.)...
First based in CAP theorem
we will require P because we require to have system that works doesn't matter if a service is down
I would say that we can predered Availability over Consistency, because we require that the system needs to be available for the cron executions
AP
The requirements says that we will require to handle thousands of tasks simultanously with minimal delay and high reliability
we will require a system with availability 99.99%
the tasks will be executed close to the programed time
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Based in the requirements we require to handle thousands of task executions simultaneously
for this case we can start with 10 million of users based on that
Daily Active Users: 1M (10%)
Each User schedules 10 tasks per day (5 one time / 5 recurrents)
Tasks created daily: 1M x 10 = 10 M/day
10M / 84600 seconds = 116 writes/second
Spike: ~5x -> 600 writes/second
Reads: ~2x Writes => 230/s
Spike: ~5x -> 1200/s
Task Execution
One Time: 5M / day
Recurrent: 1M x 5 x 1/day = 5M day
10 M executions /day -> 116 executions per second
It's really probably that the executions will be in groups of time: midnight, start of an hour on intervals of 15 or 30 min, so we can have spikes of 10x-50x
Execution Spike ~1,000 - 5,000/s
So it's require to be available to handle this
Storage:
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
The base paths at a client/request perspective will be:
Create a new one time Task POST /api/v1/tasks
payload:
Response:
Create a new recurring Task POST /api/v1/recurring-tasks
payload:
Response:
Cancel task PATCH / api/v1/tasks/{id}/cancel-task
Response:
List Pending tasks GET /api/v1/tasks?status=pending
Response
Get task by id GET /api/v1/tasks/{id}
Response
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
first will be a basic structure for the request management, using Api Gateway for request /resource management
Load balancer as well for basic request management for the multiple server instances to have better availability and don't send a request to a death service
rate limit for basic API security to avoid multi request attacts
for the one time task creation:
an active task service will received the request
Will save it in to the database the single task register
for the recurrent task creations:
an active task service will received the request
we will save the recurring task register with the calculated next iteration value
during the creation of the execution time, in order to avoid issues like multiple cron tasks executed at the same time (midnight for example) we can use the algorithm that helps to set the execution time with small differences of time between them
The cron service will be checking the database finding by the next pending tasks to be executed
We will be receiving the tasks to execute in batches
once we get them, those are going to be added to a QUEUE, I'm thinking in something like kafka because I required event distribution
once the task is in the queue we are going to have different workers to handle those tasks, so if it's required during the spikes we can add more workers as required
The worker is going to update directly the status to something like running in order to make the tasks idempotent, so if something is already running and a workers receives it, we can just ignore it
The dispatcher will be the one that is going to directly execute the task, and once is finished we can set the status to finished or set a failed status if something fails during the task execution
on each
Also we will have a logging service, that will be the one that is going to manage all the information related to the historical execution status
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
We will have multiple base entities:
I'm thinking to have as source of true a SQL database like postgres or a cassandra db for save all the entities
this in order to have strong consistency for the registers
handle well defined relations
But, I will be using CQRS to make this a little bit more faster in read having a strong write consistency
For the task_logs I will have a read replica in Elastic Search in order to make more Easy the Audit Logging requests because this table can have hundred of thousends of registers in a single day, so I will need to receive that data quickly
to make more quickly the query that will be handling the cron job, I will have two scenarios:
These as well will be saving in elastic search for less latency
we will be managing a life time for the tasks inside elastic search
I will be talking about the 2 points that I think that are the more important ones
For the first one, to avoid thundering herd problem, I would spread executions using jitter. Instead of scheduling thousands of tasks for the exact timestamp (e.g. 12:00:00:000) I would introduce a small randomized offset when computing next_run_at and also scheduling the next_retry_at. This distributes the workload across a short time window, reducing load spikes on the scheduler, workers, downstream services and the database
for the second point:
The scheduler continously looks for tasks whose next_run_at is less that or equal to the current timestamp and the status is still PENDING. Postgres is the source of truth, so task selection always happens there to guarantee consistency. ES is only for querying and auditing completed executions
I wouldn't use Elastic Search for this decision because Elasticsearch is eventually consistent. A recently created task might not be indexed yet, cousing the scheduler to miss its execution. Elasticsearch is much better suited for searching historical executions, monitoring dashboards, and audit logs
Once the scheduler identifies the due tasks, it publishes lightweight execution events into kafka. At this point the schedulers responsability ends. Kafka acts as the buffer between schedulling and execution, allowing both components to scale independently
For the worker instances consume execution events from kafka. Before executing any task, the worker performs an atomic clain to ensure that only one worker owns the execution
this can be implemented with an SQL update of status with where status = PENDING status, if there is no updated values, we can say that another worker already took the ownership of the execution and the execution will be skipped
Instead of relying on long lived database locks, I would use a lease mechanism. Every running task has a lease expiration timestamp that is periodically renewed while the worker is alive. If the worker crashes during execution, the lease eventually expires, allowing another worker safely reclaim and continue processing the task. This will prevent tasks from remaining permanently stuck in the RUNNING status
for the task execution, after succesfully claiming the task, the dispatcher invokes the appropiate executor based on the task type. Since failures are expected in distributed systems, I would assume that every execution can happen more than once
Because of that, every execution should include an idempotency key, allowing downstream services to safely ignore dupplicated requests. this protects the system in situations where the business operation succeed but the worker crashes before persisting the successful result
for the logging service will be saving each status transition, such as pending, running, completed, failed or retrying, this is emitted as an immutable event and indexing into elastic search
this gives us a complete audit trail, enables operational dashboards, and allows engineers to debug execution history without impacting the transactional workload in postgres
There can be a good option for solve some latency issues related to the list of tasks that are going to be executed inside the scheduler
The first solution can be use Redis with a ZRANGESCORE
and instead of that the scheduller will get the list from postgres, we can get it directly from redis
and the cache population will be during the task creation
postgres will be keeped as source of truth
If redis becomes unavailable, schedulters temporarily fall back to postgres pooling. once redis is available again, we will need to rebuild from the source of truth