Requirements

Functional Requirements:

• Users can query current weather conditions by city, zip code, or exact coordinates.
• The system provides customizable weather forecasts (e.g., hourly, 7-day).
• The system sends push notifications or live data alerts for severe weather changes.

Non-Functional Requirements:

• High Availability (99.99%): Weather data is critical; downtime is unacceptable.
• Low Latency (< 200ms): Read operations must be lightning-fast.
• Scalability: The system must handle extreme, sudden traffic spikes (e.g., during hurricanes).
• Fault Tolerance: The system must resiliently handle external vendor outages without cascading failures.
• Data Integrity: The system must guarantee the accuracy and freshness of the ingested meteorological data.

API Design

We add a dedicated search endpoint to resolve ambiguous location queries into exact coordinates/Geohashes before hitting the weather endpoints.

• Search Locations (Geocoding):
• GET /v1/locations/search?q={query}
• Response: 200 OK [{ "city": "New York", "country": "US", "lat": 40.7128, "lon": -74.0060, "geohash": "dr5re" }]
• Get Current Weather:
• GET /v1/weather/current?geohash={geohash}
• Response: 200 OK with standardized JSON payload.
• Get Forecast:
• GET /v1/weather/forecast?geohash={geohash}&days={n}
• Subscribe to Alerts:
• POST /v1/alerts/subscribe

High-Level Design

To achieve our low latency, high availability, and newly added reliability goals, the architecture is decoupled into distinct read, write, and validation paths.

• Edge & API Layer: A CDN caches common API responses, while an API Gateway manages routing and rate limiting.
• Read Path (Weather Service): A heavily scaled microservice that prioritizes a distributed in-memory cache (Redis) before falling back to the Time-Series Database.
• Write & Validation Path (Ingestion Service): Dedicated workers pull data from 3rd-party meteorological agencies. This path now includes robust validation and circuit-breaking logic.
• Event Bus (Kafka): High-throughput message queue for streaming ingested data.
• Processing & Alerting: Consumer services update databases, refresh caches, and trigger notifications for severe weather.

Detailed Component Design

1. Managing Popular-Location Traffic (Thundering Herd Protection)

• The Problem: If the cache for a massive city like "London" expires during a storm, millions of concurrent reads could bypass the cold cache and hammer the Time-Series DB (TSDB) simultaneously, causing a crash.
• The Solution (Request Collapsing / Distributed Locks): We implement a distributed lock mechanism in Redis. When a cache miss occurs for a Geohash, only the first request acquires the lock and queries the TSDB. All other concurrent requests for that same Geohash wait a few milliseconds and then read the freshly populated cache. Alternatively, we can use a Background Refresh strategy where worker nodes preemptively refresh the cache for the top 10,000 most popular cities before their TTL expires.

2. Data Normalization & Consistency

• The Problem: NOAA might return { "temp_f": 72, "wind_mph": 10 } while OpenWeather returns { "temperature": { "value": 22.2, "unit": "C" }, "wind": { "speed": 4.47 } }. Passing these raw payloads downstream causes data fragmentation.
• The Solution (Normalization Layer): Immediately after ingestion, a Normalization Service maps all external data into a strict, unified internal schema (e.g., all temperatures in Kelvin, all speeds in m/s, standard status codes). Only normalized data enters the validation phase and Kafka pipeline.

3. Search & Geocoding Service

• We introduce an independent Geocoding Service backed by a spatial database (like Elasticsearch or PostgreSQL with PostGIS). This allows users to type "Springfield", lists the possible matches, and returns the unified Geohash. The frontend then uses this Geohash to query the Weather Service, vastly improving cache hit rates since queries are standardized to grid coordinates rather than arbitrary strings.

4. Resilience & Outage Mitigation (Retained from previous iteration)

• Circuit Breakers & Stale Cache Fallback: Protects the system from vendor outages by extending the TTL of existing cached data rather than failing completely.
• Multi-Source Consensus & Anomaly Detection: Drops unrealistic data spikes (e.g., faulty sensors reporting 150°F) before they enter the system.