Requirements

Functional Requirements:

• Allow users to query current weather conditions (temperature, humidity, wind, etc.) for a specific city, coordinates, or region.
• Provide weather forecasts for upcoming days (e.g., 7-day or hourly forecast) so users can plan ahead.
• Enable users to look up weather by city name, zip code, or latitude/longitude coordinates.
• Push notifications, alerts, or live data updates to users when weather changes or severe events occur.

Non-Functional Requirements:

• Avability over consistency, its better to serve stale data then none
• Latency over consistency, its key to be fast
• Scalability, we need to scale to a wide variety of users

API Design

1. get_weather(longtitude, latitdude, timespan)
2. subscribe_to_weather_events(longtitude, latidude)
3. get_coordinates(): this takes in a city name, zip code, or other geo-space format and returns longtitute and latidude coordinates. this way the other apis only work with longtitude and latidude

High-Level Design

lets assume we have 10 million users & 1 million DAU users

each user asks for the weather once, means 1 million queries per day, and about 10 queries per second (72k seconds in day round to 100k seconds in day)

Another issue is that this traffic will have surges. When a storm or other weather event occurs people will use the app much more. We should use scalable infastructure in the cloud so that we can scale services & caches & the number of read replicas horizontally.

we can assume we get our weather data from a few external providers. If our providers update us once every minute with the latest weather data, then we are doing 1 write for every 600 reads. Goal will be to heavily optimize for reads.

User will interact through a mobile app or website and make a request like what is the weather in Miami.

1. we have a coordinate service that handles the get_coordinates() api
2. then we translate this request to a request to our main service
3. this service either returns a cached response or queries our database for the information
4. because of the scale of traffic & need for caching, we will likely have a load balancer around a cluster of servers & have an explicit redis or memcache. We may even want 2 layers of caches to get the read optimized performance we need

When a user subscribes to notifications, we will write that down to a table. Then we will have a service that reads database for new weather alerts, when it finds one it will use a notification service to send SMS, emails, or push notifications to the registered users.

When we get data from external providers this we might either 1. recieve a stream of events or 2. poll their apis for the latest data. We will have a weather_data_extraction_service that constantly polls for this info, transforms the providers schemas to match ours, and stores them in the database.

Since these weather logs do not have relationships, we can safely use nosql like a key-value dynamo database like dynamodb

Detailed Component Design

Weather service caches:

with 600x more reads than writes caching is critical. First we want to use a Content Delivery Network to make sure that users are using a server that is locally near them. This is 1. to reduce latency & also 2 becuase users will typically make localized requests, asking for the weather of the city they already live in. Each localized CDN node should have a cache, like Redis, to serve popular requests quickly from memory. If it doesn't exist in the cache then it can fetch the data from the weather service and add the value to the cache. Since weather data is updated at a regular cadence we can set a TTL to items in the cache, perhaps every 15 minutes. At that TTL the cache item will be updated & refreshed. One key thing to prevent is a cache stampede, if many caches recieve the same request because of a weather event, or the TTL expires at the same time. We should be sure to hold locks & coalesce cache refreshes to prevent this

Weather data extraction service

We will get our data from external providers we do not control. We will want to constantly refresh at a regular cadence, while not incurring too much cost from unnecessary requests or hitting limits. Since we do not control providers we will want to have one service that handles reading the provider info & processing them into our database schema. While we might extract the data from an api or event stream, webhooks etc. we probably want to translate them all into a real time event stream. We can then manage those with something like kafka. Therefore this service will do the job of reading events, and translating them into real time events. Use round robin to make sure we do not reach limits with any one provider. Then we can have a translation_service that consumes these messages and stores them in our database in our schema. If a provider goes down we simply do not send any updated events, or if we have suffiicent providers, we may have a backup one that we use instead. Regardless we simply let the application run with stale data if necessary. The rate of traffic here will not scale with users, only increase if we increase the frequency of updates or add addditional data providers. Because of this we can reserve a specific amount of computes or use local hardware & do not need to rely on dynamic scaling.

Weather events

Users can subscribe to weather events. We will maintain a database table recording who is subsribed to waht areas events. The notification sevice will

a. scan the latest updated weather database rows for events. There is a lot of business logic here about what qualifies as an event.

b. if a new event is created, will enqueue messages to send notifications to all subscribed users. This is fanning out to potentially massive numbers of people, hence the queue.

c. These messages get processed by a dynamically scaled service that then sends the appropriate notification. This will vary wildly in traffic size

Key consideration is that the notification service itself will have a predictably sized workload but the message processor will need dynamic scaling.