Loading...
GET /api/v1/weather/current -> WeatherInfo, 200 OK
{
city_name?
zip_code?
lat?
lang?
region?
}
GET /api/v1/weather/forecast -> WeatherInfo[], 200 OK
{
city?
lan?
lat?
region?
}
GET /api/v1/geocodes/{name} -> Location{name, zip_code, lan, long}, 200 OK
POST /api/v1/weather/subscribe -> 200 OK
{
city_name?
zip_code?
lat?
lang?
region?
}
POST /api/v1/weather/unsubscribe -> 200 OK
{
city_name?
zip_code?
lat?
lang?
region?
}
since we want low latency we'll use a CDN to return caches API responses quickly to common queries, with a low TTL value so data will remain fresh.
we'll use an API gateway to handle user authentication and authorization, load balancing, TLS termination and rate limiting.
we have a weather service to supply the main functionality of the application.
we'll also use a 3rd party weather provider to get information about the weather - which is used by the weather service instances.
we'll also use a subscription service to handle subscribing or unsubscribing to a specific location.
another component is a stream service, like kafka, to group subscriptions based on location to server as consumer groups.
another component is a notification service to notify users about weather changes or alerts they should be aware of, based on the consumer groups they are in in the stream.
we'll use a cache to keep data about highly requested geocoding mappings, weather information and forecasts.
and finally we'll use a relational DB to store the geocoding mappings, weather info and forecasts.
when a user requests current weather(with predefined data like temperature, humidity, wind speed, etc.) it reaches the CDN which calls the API gateway that directs the request to an available weather service. the weather service checks the cache for the city name/region/zip code mapping to lat and long coordinates and then checks if data exits. if it exists it returns it. if not, it checks the DB for the current data(not old, stale data) with slight delay - if it exists the service populates the cache and returns the response. in case the data doesn't exists at all it uses the weather provider to fetch the data, populates the DB, and the cache, and only then returns a response to the user.
when a user requests a forecast it works in a similar fashion - trying to fetch the data first from the cache, then from the DB and otherwise from the weather provider, populating the DB and cache when information is missing.
when a user subscribes or unsubscribes it is injected by a subscription service that does 2 things: adds a subscription/removal to the relevant stream and write the subscription to the DB that will serve as a source of truth.
for things to run ever faster we'll introduce another service weather update CRON jobs - these will query the weather provider on a regular basis in order to populate the DB with current and future data(forecasts) to avoid waiting for the weather provider every time and to act as a publisher for weather changes or severe events that needs to trigger notifications/alerts. they also invalidate the cache for the relevant data.
when a user subscribes/unsubscribes to a specific location updates the subscription service updates DB for that specific location and then the suitable consumer group. when fresh data comes from the weather updater CRON jobs, the if there is something that triggers a notification, for example higher temperature than usually at a specific location, the weather updater creates an event with the relevant details. the event is consumer by the specific consumer group which triggers the notification service to send a notification as a SSE to each of the users in the relevant consumer group.
cached data is only relevant for a short period of time depending on the type of information:
when ever data is fetched from the DB, it is fetched based on the location the user requested and the timestamp, in order to fetch the latest information possible. meaning using a geospatial index.
fresh data is achieved using the various caching along the way:
another aspect to keep the data fresh is using the weather updater CRON jobs - we can using multiple of these to query the weather provider and scale them to update horizontally(instead of having a few that are in charge of multiple regions, each could be in charge of smaller areas).
the various services - subscription, weather service and weather updater can all be scaled up or down depending on CPU usage to fit an increasing or decreasing amount of users. so in the case of increased usage like a storm or hurricane, where people check the data a lot servers can scale.
we have a single weather provider which is a single point of failure - we can introduce more providers and use an adapter pattern to spread the load among multiple providers to not relay on a single provider.
we can also add circuit breakers in front of each provider to allow them to recover gracefully and meanwhile relay on information in the cache and DB - a bit stale but we don't want to keep a provider down. when a provider is down we trip after 5 consecutive failures within 30 seconds and return cached data for 60 seconds, implementing a half-open probe after 30 seconds to check if the vendor is back online. meanwhile we try a different vendor if possible, otherwise we relay on data in cache and DB. the communication with vendor/s return once a health check is performed and they function properly.
the system is read heavy and the users expect fresh accessible data:
so data is highly available - in a CDN for frequent API responses, cache for sessions, geocoding mappings, current weather and forecasts, DB with geospatial indexes.
the data is eventually consistent using a request miss(not in cache or DB) which makes the weather service query the weather provider. stale data is viewed for a grace period of 10 or 15 minutes before updating again from the DB that should be updated.
in a case where a weather provider request fails or sending a notification fails we perform retries with timeouts with exponential backoff with jitter. the worst case here is slightly stale data but for being highly available its a relatively graceful solution.
the data that is inserted to the DB should be idempotent based on the location and timestamp. if any requested data is received and fresh data is already exists it is discarded since its not needed any more. duplicated data for the same request are also discarded. that way the information remains fresh.
another possible improvement is to separate the DB to 2 DBs - a relational DB(Postgres) for user data and subscription. and a NoSQL DB(Cassandra) for the weather data and forecasts to handle growth of data entries. each entry is roughly 100 bytes(id, lat, long, temp, humidity, wind speed, etc., timestamp), lets say there are roughly 30K relevant locations and we update a locations data at least 4 times an hour - thats roughly 0.5Gb per day. and we do the same for a weeks forecast so another 3Gb. that's 3.5 Gb data per day, which is roughly 1.5Tb per year. so we'll move only the raw responses from the weather provider to a NoSQL DB that can also handle high traffic and support scaling effectively.
the relational DB can easily handle the data that remains there - subscriptions to events (user id, lat, long, status, createdAt, updatedAt - roughly 200b) and user info(roughly 500b) and mappings (lat,long, name - roughly 200b) - data that changes infrequently - 5 * 100M* 200b + 500 * 100M + 60K locations * 200b, which is roughly 2Tb.
request coalescing will happen in the weather service, when receiving a multiple of similar requests based on the location it would prioritize handling those using a single flight request to the cache, DB, provider and answer all of them.