Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
Number of incoming tasks per second = 1K approx
~1K tasks executing in the system per second
~40% tasks are recurring. So during idle time or low traffic, ~400 tasks are running on the system.
Execution memory required by a task = 250 MB average.
So max runtime memory required = 250 MB * 1000 = 250 GB. This is the runtime memory required on the compute infrastructure.
Database
=========
Size of each task object = 50 MB
Size of status object = 500 bytes
DB requirement of new non-recurring tasks per day = 600 * 50 * 10^5 = 3 GB
DB requirement of new recurring tasks per day = 400 * 50 * 10^5 = 2 GB approx
Define what APIs are expected from the system...
POST https://
JSON Request
{
task_name:
is_recurring:
interval:
priority:
lambda: ------> indicates the lambda function to execute.
}
Returns 201 OK and unique taskId
GET https://
Returns a single JSON object or array if recurring
{
task_name:
start_time:
end_time:
status:
additional_info:
},
{
.....
}
DELETE https://
Returns 201 OK
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
Task DB
=========
Both Read and Write load on the Task DB is equal in this system.
Task information has a well-defined structure and access pattern for Task DB is based on Task Id key. So using a relational DB for Task DB alongwith sharding using TaskId is appropriate.
Every task has a state at any given time. States are:
CREATED
STARTING
RUNNING
COMPLETED
FAILED
Task Table
==========
TaskId --- Primary Key
Priority
Task Name
isRecurring
RecurringInterval
Status
LambdaToExecute
CreatedBy
CreationTime
Task Status Table
=============
TaskId -- Primary Key
Start Time -- Primary Key
End Time -- Primary Key
Status
More Info
Analytics DB
==========
Analytics DB is used for studying task behavior, patterns, monitoring etc. It is a columnar NoSQL database such as Cassandra or BigQuery.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
User Request: Incoming requests add to a priority queue for tasks.
Priority Queue: A queue for incoming tasks according to priority. Multiple queues for pre-determined intervals of recurring tasks.
Workers 1 -N: Workers for each Priority Queue. Number of workers can be added/reduced for scalability. They pick up tasks and hand over for pre-processing.
Preprocessor: Basic validation + any pre-processing logic. Verifies that the time to re-run a scheduled task has been reached. Also re-adds the task to queue for re-scheduling if recurring. Updates task state to 'STARTING'. Adds to execution queue
Execution Workers 1 - N: Pick up tasks from the execution queue, change task state to 'running' and send them to compute infra.
Compute Infra: Infra containing CPU, I/O resources and machines on which to run the task. Once complete, it updates the task state to COMPLETED.
Logging Infra, Dead Letter Queue, Monitoring/Alerting - Execution progress and output of a running/completed task is pushed to logging/monitoring/alerting infra. In case of failure, it is added to a Dead Letter Queue.
Analytics DB: Stores analytics about completed, failed tasks.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
User Request: Incoming requests add to a priority queue for tasks.
Priority Queue: A queue for incoming tasks according to priority. Multiple queues for pre-determined intervals of recurring tasks.
Workers 1 -N: Workers for each Priority Queue. Number of workers can be added/reduced for scalability. They pick up tasks and hand over for pre-processing.
Preprocessor: Basic validation + any pre-processing logic. Verifies that the time to re-run a scheduled task has been reached. Also re-adds the task to queue for re-scheduling if recurring. Updates task state to 'STARTING'. Adds to execution queue
Execution Workers 1 - N: Pick up tasks from the execution queue, change task state to 'running' and send them to compute infra.
Compute Infra: Infra containing CPU, I/O resources and machines on which to run the task. Once complete, it updates the task state to COMPLETED.
Logging Infra, Dead Letter Queue, Monitoring/Alerting - Execution progress and output of a running/completed task is pushed to logging/monitoring/alerting infra. In case of failure, it is added to a Dead Letter Queue.
Analytics DB: Stores analytics about completed, failed tasks.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Use rate-limiting at the user input layer to prevent overwhelming the input queue. Another option is to decouple the user input from internal system by adding a message queue like Kafka. A persistent queue on Redis can reduce complexity of managing a Kafka infra.
Apache Flink can be used for processing incoming stream of tasks.
Use pre-determined intervals for recurring tasks. This makes the design simple because we already have task 'priority' for both recurring and non-recurring tasks. Each priority queue is stored in a persistent Redis cache and has its own set of workers.
Use connection-pooling to avoid opening and closing DB connections frequently.
Compute infra can be on a public cloud or internally managed. Having it on public cloud can increase operational cost if the system is used heavily. It also has security concerns and no control on the kind of hardware available (if this control is required). Running it internally can be complex to manage and operate but provides more flexibility and cost benefit in the long run. In this design, we keep the compute infra in-house.
Add service discovery component like Apache Zookeeper to identify and manage the compute infra.
The system can be deployed as a microservice where input, preprocessing and execution can scale and deploy independently.
Shard database and workloads based on task id. Use consistent hashing to implement this sharding.
Explain any trade offs you have made and why you made certain tech choices...
Use rate-limiting at the user input layer to prevent overwhelming the input queue. Another option is to decouple the user input from internal system by adding a message queue like Kafka. A persistent queue on Redis can reduce complexity of managing a Kafka infra.
Apache Flink can be used for processing incoming stream of tasks.
Use pre-determined intervals for recurring tasks. This makes the design simple because we already have task 'priority' for both recurring and non-recurring tasks. Each priority queue is stored in a persistent Redis cache and has its own set of workers.
Use connection-pooling to avoid opening and closing DB connections frequently.
Compute infra can be on a public cloud or internally managed. Having it on public cloud can increase operational cost if the system is used heavily. It also has security concerns and no control on the kind of hardware available (if this control is required). Running it internally can be complex to manage and operate but provides more flexibility and cost benefit in the long run. In this design, we keep the compute infra in-house.
Add service discovery component like Apache Zookeeper to identify and manage the compute infra.
The system can be deployed as a microservice where input, preprocessing and execution can scale and deploy independently.
Shard database and workloads based on task id. Use consistent hashing to implement this sharding.
Try to discuss as many failure scenarios/bottlenecks as possible.
Use rate-limiting at the user input layer to prevent overwhelming the input queue. Another option is to decouple the user input from internal system by adding a message queue like Kafka. A persistent queue on Redis can reduce complexity of managing a Kafka infra.
Use connection-pooling to avoid opening and closing DB connections frequently.
Add service discovery component like Apache Zookeeper to identify and manage the compute infra. It will take care of identifying failed nodes, leadership loss and re-election, sharding etc.
The system can be deployed as a microservice where input, preprocessing and execution can scale and deploy independently.
Shard database and workloads based on task id. Use consistent hashing to implement this sharding.
Database is replicated to protect against loss on DB failures.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
Support custom time interval for recurring tasks.
Identify and implement metrics to monitor the load and throughput of the system.
Pro-actively predict high incoming tasks or high load using ML algorithms on historical data.