Loading...
- User can schedule a one-off task or recurring task
- Examples of tasks are: send notification email, transfer money,
run a report and send the result.
Reliability
- Low Response Time
- Scalable: 1000s of tasks executed at once
- 100K DAU
- 100 tasks per day per user
- 10M tasks / day
- 1KB storage for each event. 10GB storage.
- Let's say 1GB is added every day. 730GB in 2 years.
-> returns event_id
Storage is within the range that fits RDB.
Consistency is important because we want to execute accurate events.
Main storage should be RDB.
- Data Model
Task Table:
- task_id
- user_id
- execution_datetime # in seconds
- recurring_period # in seconds
- type # email, payment, report
Normalization gives better consistency, e.g., if user wants to change payment amount across the board.
many-to-one from Payment, Notification tables to Task table
Payment Table:
- user_id
- task_id
- sender
- receiver
- amount
Notification Table:
- user_id
- task_id
- sender
- receiver
- message
It starts with a client making a request, which first hits the API Gateway. The API Gateway acts as the entry point for all client requests, handling initial processing and routing.
The request is then forwarded to a Load Balancer. The Load Balancer's role is to distribute incoming network traffic across multiple servers to ensure no single server bears too much load, thereby increasing reliability and responsiveness.
The Load Balancer routes the request to the Scheduler Service. The Scheduler Service is responsible for managing and scheduling tasks based on the request details.
Once a task is scheduled, the Scheduler Service places the task details into a Message Queue. This queue acts as a buffer and helps in managing the task execution flow without losing any tasks during high load periods.
Tasks from the Message Queue are stored in a Relational Database. This ensures that all task information is persistently stored and managed systematically.
(Data Durability: Sending tasks from the Message Queue to an RDB before they reach the Task Service ensures that tasks are stored persistently. This is crucial for data durability, meaning if a failure occurs in the Task Service or elsewhere downstream, no tasks are lost since they are safely stored in the database. Recovery and Replay: In case of system failures or maintenance, having tasks stored in an RDB allows for easy recovery and replay of tasks. The Task Service can re-fetch the tasks from the database and reprocess as necessary.)
The Task Service retrieves task information from the Relational Database. It processes these tasks based on the stored procedures or task details.
(We could modify the database schema design with one column shown new, processing, completed, failed to get the new tasks for execution.)
After processing, the Task Service may generate further actions such as sending emails or processing payments. These actions are placed into another Message Queue (Message Queue 2) to handle these operations.
The tasks in Message Queue 2 are then distributed to specific services like an Email Sender, which handles sending emails, and a Payment Sender, which handles transaction processing.
The Email Sender, Task Service, and potentially other services interact with a Cache. The Cache is used for storing and quickly retrieving frequently accessed data to improve the performance and responsiveness of the system. (The email sender may cache the most recent emails, or email templates from the Cache. The payment sender may cache the payment API URLs. Tasks could get the task status, configurations, results, and the task services in the system. Schedule service can quickly get to the task lists, time information, etc. It is important to make the cache stay updated with the RDB.)
(Email and payment are two tasks defined here)
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
I would imagine a message queue has a functionality to run task at a certain time.
If there is no such functionality, I would implement it using a topic.
I can create a topic that runs at a certain time, e.g., May 1st, 9:00AM PDT.
Task handlers such as Email Sender or Payment Sender subscribes to this topic and execute tasks.
- Monitor all components for response time and resource situation.
- For example, if Email Sender or Payment Sender lags behind, notify sysadmin immediately.
We chose Relational DB as the main storage for tasks.
Storage size is within the range that fits RDB.
Consistency is important because we want to execute accurate events.
Main storage should be RDB.
NoSQL DBs give better scalability. However, consistency is the strength of RDB.
RDB cannot scale horizontally. Therefore, we need to partition data by user_id. This gives good locality. e.g. if user sends 1,000 events, metadata can be read from cache.
- Notification or Payment fails. Retry N times using Message Queue. After that, notify the user of failure.
- Payment does not go through for the lack of fund or authorization. Cancel event and inform the user.
- RDB cannot scale. Partition data by user_id. This gives good locality. e.g. if user sends 1,000 events, metadata can be read from cache.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?