Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
List the key non-functional requirements (performance, scalability, reliability, etc.)...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
Here are some APIs:
For users to view the weather of a location:
v1/view_weather {
consumer_id: int,
cityName: String,
geoLocationId: int,
}
For this API, we can use either a cityName or a geoLocationId to search for current/predicted weather for a given location. We will return a json blob of data, including temperature, humidity, air cleanness index etc. The response will look like:
{
geoLocationId: Int,
weatherMetadata: {
[
date: String,
min_temperature: int,
max_temperature: int,
humidity: int,
air_quality: int,
},
......
]
For users to search for a location:
v1/search {
search_term: string,
filters: List
limit: int,
For this API, we will take in a search_term from users, we will do both autocomplete as well as full text search. We will return a list of cities that users can choose to view the weather for. It will look like:
city_names: [...]
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
How do users see the current and predicted weather for their current location?
On client side, we automatically read user's GPS coordinates, and sends that as a parameter in the API request. In weather service, we do a look up by city for the user's GPS coordinate. If the user hasn't enabled GPS on their device, we will ask them to manually input their city.
On the write path, the current weather, including temperature, humidity, wind etc. for all cities in the world, are being aggregated from official weather information throughout the world in weather ingestion service, it writes weather information to a kafka queue, which is consumed by database writer, and writes the weather information to a noSQL database.
On the read path, after authentication, rate limiting and load balancing, we call the weather service. The weather service does 2 things. First, it reads the current weather for the requested city from the database, with a caching layer. Then, it calls a machine learning model in prediction service, to predict the weather for that city for the next 7 days, given current weather conditions. We then return current weather plus predicted weather for the next 7 days to the user.
How do users search for other cities?
When users want to view the weather of another city, they can start typing in, and we will do prefix matching in the ElasticSearchIndex. We will return a list of cities that match with the prefix. When users actually select a city, we will call the weather service to get the weather data for that city.
Choice of storage:
In the database, we will be storing current weather data for all cities. A flexible schema could be:
{
current_time: timestamp,
city_name: string,
country_name: string,
weather_metadata: json blob
}
For database, we will be using a noSQL database like cassandra. Given our system will handle heavy writes from Kafka queue, and we want to have a flexible schema for the weather metadata. Also, we won't be having complex relations and joins. For the database, given we will store weather information for all cities, with timestamp, we will be sharding the database by city_name + country_name with a consistent hashing strategy.
We add a cache layer (Redis) on top of it to improve performance. It will be a read through cache, where current weather data for a given city has a TTL of 1 hour. This way, we improve read performance and reduce latency. A TTL of 1 hour reduces stale data problem for the weather use case.
Choice of prediction model:
For prediction, we will be training a neural network model, trained on past weather data for all given cities, and offline evaluated and tested. The model will be deployed on Kubernetes in the prediction service.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
How is weather data ingested into the system?
In the weather ingestion service, we have a pre-configured list of all included cities for all included countries in the world. Then for each city, we set up webhook with their corresponding weather web server. In case of any weather update, it is pushed to the weather ingestion service. Here, a push strategy works better than pull, since we don't need to constantly query the official weather server of each city. Upon receiving an update, we trigger an event on the Kafka queue. If a city doesn't have a weather reporting service anywhere, we skip that city in our weather service as well, given there is no data. If inconsistent weather information is received for a city, we send an API request to the weather server of that city, and try to reconcile. If no reconciliation can be done, we send a Kafka event indicating that the current weather for this city isn't available. For any failed messages on the message queue, we do 3 retries. Each message has information of: timestamp, city, country, weather metadata. Upon receiving a duplicate message, we won't be making updates to the database.
On Kafka message queue, we have a filter that flags any events which contains extreme weather. For such events, we send the event to notification service as well. Upon consumption of these events, notification service will call a third party notification provider to notify the client.
How do we handle concurrent QPS of 12000?
There are several steps in scaling the system:
In API gateway, we implemented authentication and rate limiting, so unauthorized users can't send requests, and a single IP can't bombard the service with requests.
The load balancer uses a consistent hashing strategy. For each request, we do 2 steps. First, we identify the geo location of the sender. Our services are deployed globally, so we map the request to its nearest server region. Then we generate a hash from the consumer_id of the request, using consistent hashing to find a designated server for the request.
On data read side, we have a scalable redis cluster as the caching layer. It's a read through cache with a TTL of 1 hour. So in peak hours, the latency is very low, and reads to the database rarely happen.
For the prediction service, we have a pre-trained neural network model deployed on kubernetes, so it can do live prediction with live data. However, we also introduce a caching layer here. If the same request (with same timestamp and city) comes, we read from the cache instead of doing a live inference.