Requirements

Functional Requirements:

• Allow users to query current weather conditions (temperature, humidity, wind, etc.) for a specific city, coordinates, or region.
  • System should automatically detect the location of the user if permission is given and provide the user weather information
  • Enable users to look up weather by city name, zip code, or latitude/longitude coordinates.
• Following would be the weather information provided
  • Temperature ( maximum , minimum and current )
  • Precipitation or chance of rain
  • Relative humidity
  • Wind
  • Weather description - windy, rainy, cloudy and so on
  • Visibility
• Provide weather forecasts for upcoming days (e.g., 7-day or hourly forecast) so users can plan ahead. Forecast will include only the following info
  • Temp range (Max, min )
  • Weather description
• Push notifications, alerts, or live data updates to users when weather changes or severe events occur.

Non-Functional Requirements:

• Response time for any weather info query should be very quick. Any query should return response to user within 1 second
• We can expect around 50000 users to request weather at the same time from various locations. Average load should be 50000 requests per second
• High availability - Our weather service should be always available and even if any service is down, we should be able to provide both location and the accurate weather information. We should aim for 99.999% uptime
• Scalability - During abnormal weather events such as earthquakes, cyclones or tornadoes weather queries would peak. One can assume a peak time load of 500000 requests per second. Our system should be capable of scaling up resources to handle that increase in load and also go back to the average load handling capacity post the peak.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

1. Location
   • /api/coordinates/{location} ( to retrieve the coordinates of a given text location )
     • Method : GET
     • Response :

{

latitude:

longitude:

}

• • /api/locations/autocomplete?q=<text>&limit=10 ( to show auto complete options when user starts typing for easy selection of location )
    • Method : GET
    • Response:

[

Bengaluru,

Benton,

.

.

]

1. Weather info
   1. /api/weatherInfo ( To get the weather information when coordinates of that location are given )
      • Method : GET
      • Request Body :

{

latitude:,

longitude:

}

• • • Response:

{

temperature:

{

max:

min:

current:

},

weather:

cloudy / rainy

..

..

}

}

b. Forecast - To provide weather forecast for the next n days

/api/weatherForecast/{n} ( To get the weather forecast for the next n days )

• • • Method - GET
    • Response body :

[

{

max:

min:

weather: cloudy/ rainy / sunny , etc.,

},

{

max:

min:

weather: cloudy/ rainy / sunny , etc.,

}

]

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

Mobile app

• mobile application used by the user to access the weather data and forecase
• It connects to the backend through an API gateway

Browser app

• A Web app that can be used to access the weather data and forecast data using the browser
• It connects to the backend using API gateway
• Users can use the browser app to check weather forecast and also see the current weather information for selected cities

3rd party providers

• Weather data is provided by multiple third party services and these represent those services
• These could be providing data for a particular country or a particular weather category like storm or snow and so on

Injestion workers

• We can't directly store the data sent by 3rd party services, so we use injestion workers which then adapt the data to our format and then push them into a kafka pipeline
• Data is pushed into kafka pipeline so that we can process the weather data in an aysnchronous manner

Kafka pipeline

• Multiple services such as forecast service, notification service and current weather service subscribe to various topics in the kafka broker and these are then pushed by the respective services into the Weather data DB

Weather Data DB

• We use a cassandra database as we have lot of writes and we have a more predictable querying pattern so we can optimise the tables according to our queries. Hence we choose Cassandra
• This will store the current weather information and also the forecast information

Redis cache

• Services will first see if the data is present in the cache before going to the DB.
• We don't blindly update every data from DB in the cache. Around 80% of the querying is for 20% of the locations. So we will have a cache eviction policy of Least frequently used combined with a short TTL on the cache. So locations which don't have that much demand won't take up precious space on the cache.

Current Weather service

• This service takes the location coordinates as its input and produces the weather information for that location as the output
• The service will query the redis cache and in case of cache miss, we will load from DB and update the cache
• This service is stateless as it merely takes the input coordinates and fetches the weather info and can be easily scaled to multiple instances to match the incoming load

Weather notification service

• This service subscribes to the emergency weather events topic, where urgent events like earthquake, severe storms, cyclone and so on. These events need to be pushed to the user as push notifications and Weather notification service will push these events to the respective users based on their locations

Location service

• This provides multiple functionalities
  • Auto complete - when given a 2 - 3 letter text, it fetches the top 10 matching cities from its trie so that the UI can populate the options
  • When given a location, it will return the coordinates from its map

Weather forecast service

• This service is queried by the users to provide a 10 day weather forecast.
• This service sources this data from the 3rd party weather providers.
• The forecast data is then fed into the kafka injestion pipeline to the topic -weather.forecast.data and this is then stored in cassandra database
• Similar caching strategies as done fro the current weather is applicable here as well
• Similar to the current weather, here also no state is maintained and hence it's very easy to scale by adding more replicas

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

Weather Data DB :

• Cassandra DB
• Following data types are stored in the DB
  • Current Weather info
    • Location_id
    • Temperature
      • Maximum
      • Minimum
      • current
    • Humidity
    • Weather
    • Sun Rise
    • Sun Set
  • Weather forecast
    • Location_id
    • Date
    • Max temp
    • Min temp
    • Weather
• We need to shard the database by location and date combination for the forecast to ensure that we are able to service everyone quickly
• We also need to replicate every shard with 2 replicas to ensure that we have adequate redundancy. Each shard should be in a different hemisphere to ensure that there is no geo latency

Kafka pipeline

• Following topics are used
  • current.weather.info - used by injestion workers to update current weather data for different cities. consumed by current weather service
  • weather.forecast.data - used by injestion workers to update the forecast data for different cities. Consumed by
  • urgent.weather.event - used by the notification service to push any urgent weather event to the subscribing devices