Loading...
60 MLN tasks over 1 hour
6 * 10 ^ 6 / 3.6 * 10/3 = 2000 TPS
Task avg = 100 ms
Task peak = 1s
CPU usage avg 5% peak 50%
memory avg 10 MB , peak 100 MB
IO avg 1 MB, peak 5 MB
concurrency = 2000 * 0.1 = 200
cpu peak = 200 * 50% = 100
memory peak = 200 * 100 MB = 20 k MB ~ = 20 GB
Define what APIs are expected from the system...
Task {
String codeS3Location;
String cronInterval;
long oneTimeExecutionEpoch;
String idempotencyToken;
}
String scheduleTask (Task task); // returns taskId
String cancelTask(String taskId);
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
We would need a table Tasks. Both tables would have a primary key taskId (to allow quick deletion) and an index on executionEpochSecond. This table could be a key-value story, giving us high availability and scalability.
TaskScheduler exposes scheduleTask and cancelTask APIs. When scheduleTask is called it persists it in TasksTable with executionEpochSeconds which gives the second of the day in which the task should be executed. TaskOrchestrator polls every second by executionEpochSeconds for a given point in the day
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
TaskScheduler scales extremely well. Each requests only does a write to the TasksTable which should be extremely fast.
Explain any trade offs you have made and why you made certain tech choices...
I could potentialyl use EventBridge
Try to discuss as many failure scenarios/bottlenecks as possible.
Rate limiting and auto scaling for TaskScheduler.
Task Execution fails - how do we notify the user? They would need an API to check the status of the executed task.
Reading the code from the customer provided bucket could fail
Autoscaling for Taskexecutor
timeouts for task executor to prevent hanging tasks to consume resources for too long.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?