Requirements

Functional Requirements:

• Users can specify a location and receive the latest weather data.
• The system has support for real-time updates where users can subscribe to receive data updates for a given location.
• Edge cases/Error Handling:
  • If the user inputs an invalid location (misspelled city name or non-existent location), the system should return a clear error message and provide suggestions for valid inputs.
  • If the latest data is unavailable the system should return cached data for the given location or notify that the weather data service is unavailable.

Non-Functional Requirements:

• 99.9% uptime/availability
• Latency Requirements
  • <2s non-cached requests
    • API calls to Third-Party Weather Data Providers
  • <1s real-time updates
    • Push updates to subscribed users upon data updates
  • <500ms cached requests
    • Retrieving cached weather data
• 10k concurrent users, can accommodate demand spikes due to severe weather events.
• Weather data is consistent and updated regularly

Capacity Estimation:

• ~100k cities -> ~1 million zip-codes
• 500 bytes per location
• 500 bytes * 1 million locations = ~500 MB
• Active storage estimates: ~1 GB - 5 GB (replication, indexes, buffer)
• Historical Weather Data: ~10 GB - 100 GB/year

API Throughput:

• 10k concurrent users
• Daily API requests: ~50 million
• Peak RPS: ~2k rps

API Design

• GET /weather/?location=<location>&units=<metric>

Response:

• POST /weather

Request:

• POST /subscribe

Request:

Response:

High-Level Design

Overview:

• User (Web/Mobile) -> API Gateway (REST APIs) -> Weather Aggregation Service -> External Weather Data Providers -> Database + Cache -> Real Time Updates

Core components:

• User (Web/Mobile)
  • Clients interact with weather data via the mobile or web application.
  • Client can specify locations and subscribe to real-time weather updates.
• API Gateway (Rest APIs)
  • REST APIs for data requests and subscription requests.
  • Cached responses/data validation.
  • API versioning and rate limiting
• Weather Data Providers
  • Adapters for third-party data providers to fetch and convert weather data to a standardized format.
• Data Store/Cache Layer
  • Cache layer (such as Redis/Memcached) for low latency responses per location (10 min TTL)
• Real Time Updates
  • WebSockets for near-real-time data updates for users subscribed to given locations
• Subscription Manager
  • Manages clients subscriptions to locations.
• Weather Aggregation Service
  • Periodically schedules jobs to populate weather data for locations (fixed interval).
  • Responds to adhoc data requests.

Detailed Component Design

Weather Aggregation Service

• Data from third party data providers are added to a message queue. The message queue is processed by service workers and the data updates make it to the Weather Service for usage.
• WeatherDataProvider for each third-party data source.
  • fetch/normalize methods implemented based on third-party data source APIs

• Data Retrieval
  • Third Party Weather Data Providers:
    • AccuWeather
    • The Weather Company
  • Third Party weather APIs support similar APIs:
    • GET /weather?location=London&units=metric&appid=API_KEY
• Data Format
  • Data from third-party data sources may have different field names (location, temperature, units). We define a standard schema that our weather service can use.
• Message Queue
  • Once data is processed for a given location from a third-party data source, it can be added to a message queue (SQS) to be ingested into the weather data store.

WeatherDataService

• API for returning detailed weather data per location.
  • GET /weather/?location=<location>&units=<metric>
  • The currently available weather data in our data store for the given location is returned.
  • If the weather data is older than some specified TTL, we can add the location to a queue in the Weather Aggregation Service to be updated. This allows our service to be responsive even if a specific Third Party Data Provider is unavailable.
• Clients may subscribe to our weather service for a given location, new updates are pushed when available.
  • POST /subscribe
• Database storage considerations
  • DynamoDB (No-SQL Key-Value Store)
    • Fast location-based (key-based) look-ups
    • TTL for auto-expiry of data
    • Scales automatically, PITR, multi-region support
  • Can consider introducing another PostgreSQL store for maintaining historical data or dashboard views
    • Looking up top 10 cities, all cities under certain weather conditions, etc
• Data format

Real-Time Update Engine:

• WebSockets in combination with publisher/subscriber pattern.
• Client's open a WebSocket connection and subscribe to updates for a given location.
• When the server receives weather updates for a given location, it pushes the update to all current subscribers of that location. This ensures we can meet our latency requirements (<1s latency upon data updates) and handle data updates during critical weather events.