Detailed Component Design
Weather Aggregation Service
- Data from third party data providers are added to a message queue. The message queue is processed by service workers and the data updates make it to the Weather Service for usage.
- WeatherDataProvider for each third-party data source.
- fetch/normalize methods implemented based on third-party data source APIs
abstract class WeatherDataProvider {
abstract fetch(location);
abstract normalize(data);
}
- Data Retrieval
- Third Party Weather Data Providers:
- AccuWeather
- The Weather Company
- Third Party weather APIs support similar APIs:
- GET /weather?location=London&units=metric&appid=API_KEY
- Data Format
- Data from third-party data sources may have different field names (location, temperature, units). We define a standard schema that our weather service can use.
- Message Queue
- Once data is processed for a given location from a third-party data source, it can be added to a message queue (SQS) to be ingested into the weather data store.
WeatherDataService
- API for returning detailed weather data per location.
- GET /weather/?location=<location>&units=<metric>
- The currently available weather data in our data store for the given location is returned.
- If the weather data is older than some specified TTL, we can add the location to a queue in the Weather Aggregation Service to be updated. This allows our service to be responsive even if a specific Third Party Data Provider is unavailable.
- Clients may subscribe to our weather service for a given location, new updates are pushed when available.
- Database storage considerations
- DynamoDB (No-SQL Key-Value Store)
- Fast location-based (key-based) look-ups
- TTL for auto-expiry of data
- Scales automatically, PITR, multi-region support
- Can consider introducing another PostgreSQL store for maintaining historical data or dashboard views
- Looking up top 10 cities, all cities under certain weather conditions, etc
- Data format
{
location: <location>,
temperature: <temperature>,
units: <metric>,
conditions: <weather_conditions>,
source: <weather_data_source>,
timestamp: <timestamp>
}
Real-Time Update Engine:
- WebSockets in combination with publisher/subscriber pattern with an implementation such as NodeJS Socket.IO.
- Client's open a WebSocket connection and subscribe to updates for a given location.
const webSocket = new WebSocket("ws://web-socket-address:port");
webSocket.onopen = () => {
socket.send(JSON.stringify({
action: "subscribe",
lon: 76,
lat: 52
));
}
- When the server receives weather updates for a given location, it pushes the update to all current subscribers of that location. This ensures we can meet our latency requirements (<1s latency upon data updates) and handle data updates during critical weather events.
const webServer = new WebServer(...);
const subscribers = {};
// Upon receiving a weather update
sockets = subscribers[<location>];
sockets.forEach((socket) => {
socket.send(JSON.stringify(update)
});