Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
1. A Business should be able to Create or update the inventory of a specific item.
2. A Business should be able to process an order for order fulfillment purposes. This includes being able to ship an item to a customer.
3. A Business should be able to receive real-time updates.
4. A Business should be able to have robust analytical tools in order to track their inventory appropriately.
List non-functional requirements for the system...
1. Consistency. The inventory should be consistent and show accurate information.
2. Scailability. The system should be able to scale with the business to ensure the business is successful.
3. Monitoring. The system should have robust monitoring tools to ensure uptime and accurate inventory tracking with reporting tools.
Estimate the scale of the system you are going to design...
Estimate the scale of the system you are going to design...
100M Users
20M DAU
20 Inventory transactions of writes/reads per day
Total transactions per day: 20 * 10^6 * 20 = 40 * 10^6 = 4 * 10^7 = 40,000,000
Data capacity of each transaction: 500 Bytes + 1000 bytes (compressed media) = 1500 bytes per transaction
Transactions per second: 40 * 10^6 / 10^5 = 40 * 10^1 = 400
Total Data throughput per second: 400 * 1500 bytes = 4 * 10^2 * 15 10^2 = 60 * 10^6 = 60,000,000 = 60 MB * 3 (data replication) 180MB per second
Total DAta per day: 40 * 10^6 * 1500 * 20 * 3 = 4 * 10^1 * 10^6 * 2 * 10^1 * 3 = 24 * 10^8 = 2,400,000,000 = 2.4 GB per day
Define what APIs are expected from the system...
https://getInventory (GET) [get all inventory items]
(user_id, timestamp, stateless_session)
https://getInventory/item_id (GET) [get specific inventory item]
(user_id, timestamp, stateless_session, item_id)
https://updateInventory/item_id (POST) [update single inventory item]
(user_id, timestamp, item_id, item_content, item_media, item_name)
https://updateInventory/BULK (POST) [update multiple items]
(user_id, timestamp, item_ids, item_contents, item_names, item_medias)
https://fulfillInventory/item_id (POST) [fulfill single inventory items]
(user_id, timestamp, item_ids)
https://fulfillInventory/Bulk (POST) [fulfill multiple inventory items]
(user_id, timestamp, item_ids)
https://Inventory/search
(user_id, timestamp, query)
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
user_table
user_id: string
timestamp: string
inventory_table
item_id: string
item_name: string
item_content: string
item_media_url: string
category_id: string FK
index: item_id
search_index: item_name
inventory_category_table
category_id: string
category_name: string
item_id: string FK
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
1. If a business is attempting to lookup a piece of media content the request first goes to the Media CDN Cache, if it's not there the request goes through the load balancer and onto the API gateway.
2. If a business is attempting to update/create inventory items the requests go through the Load balancer which evenly distributes the traffic to the API Gateways(s). The API gateway handles authentication, monitoring, and security mechanisms such as rate limiting to ensure our system does not crash.
3. The API Gateway performs routing by looking at the URL params and forwards the authenticated requests to the correct service depending on its availability.
4. Users who attempt to create or update an inventory service are routed to the inventory service which in turn routes the request to the Coordinator service. The coordinator service routes the request to the proper DB. If it's a read request it will check the inventory cache otherwise it will send the request to one of the read databases.
5. Media content is not stored in the database but rather in the Object storage database and instead a URL to the object is stored in the database.
6. Write requests are routed to the primary write database which utilizes 2 phase commit and database record locking to ensure we meet the requirements of isolation and consistency.
7. Write requests don't wait for the server request to be returned, because once an inventory item is updated the websocket request is informed and broadcasts a message to all sessions that are looking at that record. This request reaches the client and the client automatically performs a refresh to receive the real-time update.
8. All write requests are sent to the Analytics service which in turn performs updates on the analytics table.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Analytics:
1. Performing analytics such as querying the database for reports on total inventory items is an expensive operation, this is why we have decoupled it into its own analystics micro service along with a analytics database. The analytics database uses a Stars and Snowflakes technique where it creates a central table called the 'fact' table. This table has many columns and foreign keys to various tables, this is called the 'star' schema. The analytics database uses column oritened storage, this is because in analytics we typically don't need the entire row such as the way other databases do, also when we add columns from tables we do so in the same order that we read them in because otherwise lookups could fail.
Inventory service:
1. When a user updates an item they do so via the API: https://updateInventory/item_id. The inventory service communicates with the Coordinator which in turn routes the request to The primary write Database.
2. The Primary write database is a NoSQL database with LSM Trees and SStables because we prioritize writes. We use the index of the item_id to find the item in our data structure. The find operation can be performed in log(n) time since LSM Trees and SSTables are structured in a way where we can perform Binary search. Because the LSM Trees are in-memory the write operations are fast.
3. Once the item is updated a message is sent to the WebSocket service. The Websocket service broadcasts a message to all clients who may be viewing the inventory item. The Websocket service is able to do this because is supports bidirectional communication. Websockets are supported by various platforms such as mobile devices and web browers. Since websockets establish a persistent connection between the client and the server they offer several advantages over http requests and responses, such as reduced overhead (less amount of header data) and lower latency (persistent connection). This makes websockets ideal for real-time updates and great for consistency. Websockets also scale well since they reduce the overhead on the overall system.
4. If a user performs a search they do so via the API: https://inventory/search. The search API uses the search index to retrieve the data from the database. Once the data is collected the inventory service it is transformed into JSON returned to the client.
Monitoring service:
1. The monitoring service is responsible for monitoring the health of the system. It collects information from the Coordinator/Orchestration service about nodes that are working correctly, the Coordinator/Orchestration service does this via heart beats. Heart beats are periodic pings to various nodes to ensure they’re accessible and functioning correctly. These pings collect the last time the nodes have communicated with the system, if a node has not communicated over a certain threshold the monitoring service will trigger a notification to be sent to the administrator and the node will be demoted and restored to working order.
2. The monitoring service also collects information from the API Gateway. For example it collects response codes, timestamps and latency about each API request and response and if for example many http 500 server codes are picked up a notification will be sent to the administrator to indicate an issue has occurred.
3. The monitoring service also collects information from the Web Socket service. When the client is signed into the system a persistent web socket connection is generated and a heart beat with an interval of every five seconds is setup. The heart beat information is passed back to the monitoring service which uses this information to ensure the client has a reliable connection to the system. If a discrepancy in the heart beat connection is picked up, a notification is sent to the administrator and the system initiates a retry mechanism to re-establish the connection. After 10 minutes the connection is terminated without a valid response from the client.
4. The monitoring service collects logs and vital information so when errors to occur a developer /system administrator is able to use that information to investigate the root cause of the issue.
Explain any trade offs you have made and why you made certain tech choices...
1. We go for eventual consistency versus strong consistency to strike a balance between scalability and consistency. For example, when we update a record in the database we utilize the SAGA pattern over Two Phase commits. Two phase commit offers strong consistency because it locks down the records in the database and waits for all nodes to perform the update before actually committing the update and deeming it successful. The issue is it’s slow because nodes could be in various regions and this is an issue for Scalability. The SAGA pattern allows us to only hold a lock for each independent execution, and the coordinator/orchestration service monitors the rest of the nodes to see if the update was successful. If an issue is picked up, the coordinator/orchestration service performs a roll back and all nodes are restored to their previous state. This allows the system to scale better since each node executes transactions independently and we have a system in-place to resolve the issue if an error is picked up. This also helps alleviate issues with phantom locks, where database record locks could be stuck and in forever pending state if for example a node has crashes or other random issues occur.
Try to discuss as many failure scenarios/bottlenecks as possible.
1. Our database utilizes single leader replication over multi leader replication so we could run into performance issues in the future if we keep scaling. This is because when there’s only one leader, it creates a performance bottleneck since all writes must be performed by this database.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
1. Implement multi-leader replication based on regions. Where we only have single leader replication in specific regions to avoid issues that can occur when we have multiple leaders trying to concurrently write to the same record. It’s likely that certain records will only be updated in certain regions due to them being used in that location.