Loading...
# Problem description
Develop a system that provides users with current weather information for a specified location
# Functional Requirements
1. A user can request the current weather from a given location
2. The wewather data includes temperature, precipitation, wind speed, direction, etc.
1. Users may be able to create accounts or not
1. Users with accounts can add favorite locations
1. Users with account can add notifications
2. Locations can be specified by different methods: gps, postal code, city name, etc.
3. The weather data arrives to the system from 3rd party systems
4. The user can subscribe and receive warning notifications
Non-functional requirements
1. 100e6 users
2. Locations minimum granularity: 1 km2
Estimate the scale of the system you are going to design...
# Data requirements:
-User entry size: ~4B+64B+64B+4B+128B=264B ~0.3KB
-User table: 100e6*3e2B=300e8B=30e9B=30GB
-WeatherMetricType entry: ~
-Weather entry: 70B
-Location count: 1Km2=>1km*1km. Earth is 40e4 Km circunference => < 40K*40K = 1600K = 1.6M
-Weather entry time granularity: 1 minute
-Weather entry count: <70B*1.6M/1minute = 100MB per minute
-Weather table size: 100MB per minute=100MBpm*60mph*24hpd~=600GBpday~=210TB per year
-Weather table size: this is a very large over estimation, as we care mostly of a tiny part of the planet, let's say 5% of its surface, thus we can potentially only store data in the order of 10TB per year.
Read/write requirements:
user: low read/writes: 10% users daily => 10e6 per day=> ~=115request per second (low)
weather: writes: <1.6M per minute ~25k rps (high)
weather reads: 10% users read 10 times per day => 10e6 * 10 => 100M per day=>~1200 reads per second (low/medium)
weather: user reads are of current data
weather: we can provide historical data, the data is aggregated by time, location, etc.
Define what APIs are expected from the system...
# Entities
User
-id: int
-name: varchar
-email: varchar
-phonenumber: varchar
-password hash: varchar
Weather Table (70B)
-timestamp: long (8B)
-gps location: gps type (8B)
-weather metric name: varchar (32B)
-value: double (8B)
-unit: varchar (16B)
# Database choice:
Users:
- (recommended) use a RDMS like Postgres, MySql. The storage load can be handled by a single server. We can add read replicas to handle availability, reliability, and low latency (having some replicas distributed in different geographic zones)
- (alternative) we could have a document or keyvalue store with semistructured data, e.g. dynamodb, mongodb. The upside is automatic scalability and response time, the downside is weak consistency
Weather:
- (recomended) use a column based OLAP storage such as ClikHouse or Snowflake. Pros: supports queries to the dataset by range of timestamp and filtered by metric and location very efficiently. Allows aggregation of data for historical. Supports very high write rates (tens of thousands of rps).
- (alternative) a keyvalue or document OLTP db store.
- (non-alternative): graphdb, search dbs
- Additionally, we can store aggregated data for data older than 1 year old for example, and delete the high granularity data from the main weathertable.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
tradeoff: normalizing the weather metric name reduces weather table size by half and facilitates localization, but increases app complexity
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?