Loading...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
GET /api/v1/GetTask/{id} - Gets a task using ID
GET /api/v1/ListTasks - List all tasks.
GET /api/v1/GetTaskStatus/{id} - Get status of a task
GET /api/v1/GetAllFailures - Gets all the tasks that failed
GET /api/v1/GetAllSuccesses - Get all the tasks that succeeded
POST /api/v1/NotifyStatus - To notify the user of the status
POST /api/v1/CreateTask?type={} - This takes in either One time type or Fixed Interval or Cron type.
PUT /api/v1/UpdateTask/{id} - Update the task based on id.
POST /api/v1/CancelTask/{id} - Cancel a task based on id
DELETE /api/v1/DeleteTask/{id} - Delete a task based on id.
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
CDN - To host the frontend and to cache the results which takes care of 80% of the read requests.
API Gateway - Routes the traffic based on the API hit to its respective service. Can be used for authentication and rate limit the requests to stop DDoS attacks.
Load Balancer: To balance the traffic load and keep the system stable.
Kafka: High throughput Queue which can hold data for a long time making the system gracefully degrade and pick up where things are left off at the time of down time.
Taks Getter - Gets the tasks and its details
Create Task - creates new tasks
Notify Status - notifies the respective subscribers about the statuses of the tasks.
Update Tasks - Updates the tasks
Scheduler service - used to schedule service and to invoke the tasks
Postgres - DB used to store the metadata of the schedules/tasks
Redis - will have a sorted set which can be used to read in o(log N) time. It is build up on Postgres. So if this goes down we can again build using the data in Postgres.
Message Queue - to decouple the service from the executors which perform the scheduled tasks.
Executors - These are the ephemeral executors which spin up at the designated schedule and perform there task and then update it in the DB after which they terminate. New executor will spin up based on the schedule lets say for cron jobs there will be next_run_at and when that time arrives a new executor will spin up.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
CDN - Can handle high traffic and is responsible for caching as well. It can handle 80% of the read requests reducing load on the DB.
Without CDN if we went with having a server for frontend we need to manually setup caching for it which is a lot of manual labour which is not a worthy tradeoff.
We are going with Postgres to store instead of NoSQL DB because the schema is fixed we will be having a fixed number of colums for the metadata for which postgres is quite suitable also for the complex queries that will be performed that postgres can handle.
For quick looks up we are using Redis sorted sets instead of just directly querying the Postgres DB is because the lookup time on Redis is O(log N) which is very efficient compared to the Postgres. Also this reduces the load on DB.
When a executor starts working on a task we will have column like status set to claimed and claimed by set to the executor id. This way we can avoid same task being taken by multiple executors.
We will have a health check service for executors so that when a executor crashes mid task it can be detected and is taken up by another executor.
We can have idempotency key for each task execution so that we can avoid duplication execution