Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
Define what APIs are expected from the system...
POST /tasks/create - this endpoint let user schedule a task with the schedule indicated in payload and task definition as a script
GET /tasks/task_id - get a log of execution by task_id
GET /users/user_id/tasks - get a log of all tasks belong to user_id
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
For user data and task metadata we can use mongo DB, shard on user_id to easily query for a list of tasks belong to a certain user. Task DB will have timestamp and status columns. Status being completed and started. Will talk about how to check for failed tasks in the sessions below.
For task definition, we can use a BLOB storage like S3 to store the task script files, we will have a S3 link in the task metadata table
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
User flow:
When a user creates a task, it goes through the load balancer and server, it saves the task definition file in S3 and creates a link, it also saves the task metadata and the S3 link in mongo DB, the user row's task column will be updated to include the newly created tasks.
Task scheduling flow:
Workers pull tasks from task DB by checking the timestamp column, pull tasks that need to be run in the next 30 min for example. We have a bunch of task queues, tasks are sorted by timestamp on the queue, the tasks closest to now is at the front of the queue. When the executor pulls task from the queue, it first updates the timestamp to 30min later let's say and mark status to started. So that in scenario where the executor fail the complete the task, another executor can picking up by checking that the status has started but never completed. If executor successfully completed the job, mark the status to completed and other works won't pick it up.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Tires:
We mentioned there will be many queues. For the executor and queue, we can divide them into different tiers with different hardware, for higher priority tasks we can send them to high tier executors maybe with more expensive hardware which can give us even better availability.
Explain any trade offs you have made and why you made certain tech choices...
Message queues:
We pick between Kafka which is one consumer per partition, where tasks can be stuck behind long running jobs
In-memory queues:
We choose to go with in-memory message broker because it allows many executors for each message broker. When a consumer idles it grabs a task. And since it's in-memory it is going to be relatively low latency.
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?