Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
A User should be able to search for a weather location and receive real-time updates for the weather
A User should be able to see a 3, 5, and 7 day forecast of weather.
A User should see accurate and up to date information.
A User should be be able to configure notifications which are then sent to the user at a certain interval
List non-functional requirements for the system...
High availability.
Low Latency.
Strong Consistency.
Estimate the scale of the system you are going to design...
Estimating 1 million DAU and 10 Million Total users
100MB per user for photos and videos and 2000 bytes per page load and the page loads 10 x in an hour
100MB * 1 million
2000 bytes * 10 * 1 millon = 20000 * 1 million = 20000 MB = 20 GB
20 GB + (100 million MB = 100 TB)
Define what APIs are expected from the system...
searchWeatherLocation(searchTerm, timestamp, user_id, device_type) [GET]
getForecast(timestamp, selectedFilter, user_id, device_type) [GET]
updateNotification(user_id, notification_Id, notification_name, interval, timestamp) [POST]
postWeatherData(sentor_id, timestamp, location, video_content, photo_content)
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
notifcation_table
user_id: string
notification_id: string
location: string
interval: string (storing number as a string for safety reasons)
timestamp: string
user_table
user_id: string
device_type: string (ipad, iphone, desktop)
sensor_table
location: string
timestamp: string
sensor_id: string
media_video_url: string
media_photo_url: string
index: location;
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Two sensors collect weather information and update our write database with the approriate content. They also communicate with the media Database to get the appropriate url for the media content.
The load balancer ensures we distribute traffic evenly.
The client can either search for a weather location via the search service and get real time weather or the user can configure a notifcation which then sends updates to the client via web sockets.
Since we're prioritizing Reads, our databases use noSQL with a B-Tree data structure.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Database
We use noSQL with a B-tree structure because noSQL is very fast compared to SQL and the B-tree structure prioritizes reads over writes. The B-tree uses a binary search tree structure behind the scenes which is typically done on a hard disk, since it's binary search it helps us achieve logarthmic time. Most users are not going to write to the database, so the performance impact on writes is okay. We use partition our databases among all three using a consistent hashing technique. A coog consistent hashing technique can help us achieve strong consistency, because if one node fails or is added, rebalancing is targed. Rebalancing is targed because data is organized via several pointeres in a circle where each database is split into thee virtual nodes and responsible for a range of data. If one of these nodes goes down or is added we need to only look at the node before and after the target node to update the range. Overall this design helps us achieve Low Latency, high availability and strong consistency.
Web Sockets
The notification service sends notifications to the user via Web sockets. Web sockets offer low latency because they create a direct connection between the client and the server. Web sockets are efficient because they don't require frequent HTTP requests and responses, repetitive HTTP headers, and security handshakes. They're especially useful for real-time applications. Web sockets are compatible on the multiple types of devices we need to support such as desktop, ipad and iphone. Web sockets also set us up for the future incase we need to support communication from the client to the server in a different manner since they're bi-directional. We're also not sending media content in the notification so since it's characters they fit will with this design.
Explain any trade offs you have made and why you made certain tech choices...
We choose to prioritize writes over reads because the client is mostly looking up weather content and not creating notifications. This is why we chose a database that uses a B-trees over SS Tables and LSM trees. SS Tables and LSM trees are better for writes because of an in-memory write table.
We choose to not use any chaches to provide accurate data. Since the weather data is constantly updating we forgo the caches and accept the performance penalty over accuracy. This is because it will impact our latency by having to make frequent server requests to our database to pull the latest weather. If we had a cache it could improve our latency a bit since reading from cache is much faster than reading from our database server.
Try to discuss as many failure scenarios/bottlenecks as possible.
The sensors could potentially overload the write database. This is because they're configured to push data automatically to the write database in a 1 minute interval. This could potentially overload the system at some point.
There's only one write database at the moment. This is a potential issue since it can be overloaded and slow down the entire system.
There's no queue between the sensor data and the Write DB.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
Add a queue between our sensors and the write database to support high availability and system efficiency. This will ensure our system does not go down.
Add a cache with a short Time to live property to take advantage of what we can. For example, if the sensors are set to update every minute, we can add a cache with an evication policy of 30 seconds. This could improve latency.
Add another write database to ensure the first write database is not overloaded.