Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
Define what APIs are expected from the system...
POST /tasks/create - this endpoint let user schedule a task with the schedule indicated in payload and task definition as a script
GET /tasks/task_id - get a log of execution by task_id
GET /users/user_id/tasks - get a log of all tasks belong to user_id
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
For user data and task metadata we can use mongo DB, shard on user_id to easily query for a list of tasks belong to a certain user. Task DB will have timestamp and status columns. Status being completed and started. Will talk about how to check for failed tasks in the sessions below.
For task definition, we can use a BLOB storage like S3 to store the task script files, we will have a S3 link in the task metadata table
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
User flow:
When a user creates a task, it goes through the load balancer and server, it saves the task definition file in S3 and creates a link, it also saves the task metadata and the S3 link in mongo DB, the user row's task column will be updated to include the newly created tasks.
Task scheduling flow:
Workers pull tasks from task DB by checking the timestamp column, pull tasks that need to be run in the next 30 min for example. Each worker has its own task queue, tasks are sorted by timestamp on the queue, the tasks closest to now is at the front of the queue. When the worker pulls task to the queue, it first updates the timestamp to 30min later let's say and mark status to started. So that in scenario where the executor fail the complete the task, another worker can picking up by checking that the status has started but never completed. If executor successfully completed the job, mark the status to completed and other works won't pick it up.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?