1) Requirements

Functional Requirements

• Show weather for a chosen location (temp, humidity, precip probability, wind, sunny/clear, etc.).
• Allow users to subscribe/unsubscribe to city-level notifications.
• Send immediate crisis notifications (storm, hurricane, high-wind, flash flood) to affected subscribers.
• Provide daily and short-term (hourly) forecasts.
• Allow per-user notification preferences (ALL vs CRISIS_ONLY, channels).

Non-Functional Requirements

• Availability & Partition Tolerance (AP): remain available during partial failures; small staleness (seconds) is acceptable.
• Scalability: handle ~1,000,000 DAU and large burst spikes for city-wide crises.
• Latency: API reads <100–200 ms (cache hit).
• Durability: persist raw provider payloads for replay/audit.
• Observability & Security: tracing, metrics, encrypted push tokens, auth for APIs.

2) API Design

All APIs: JSON over HTTPS; standard error envelope:

1. Subscribe

Request:

Responses:

• 201 Created (or 200 OK if updated)
• 400 Bad Request, 404 Not Found (city), 409 Conflict (semantic)
• Notes: idempotent on user_id+city.

2. Unsubscribe

DELETE /v1/unsubscribe/{city}

Responses: 200 OK, 404 Not Found.

3. Read Forecast

GET /v1/weather/{city}/{YYYY-MM-DD}

Errors: 400, 404, 500.

Operational notes: cache-aside pattern for Redis; fallback to canonical store.

Operational controls

• Rate limiting (per user / per API key); 429 with Retry-After.
• Clients should retry idempotent writes with exponential backoff.

3) High-Level Design

Architecture summary

• Ingestion/Normalization → Message Broker (RabbitMQ) → Delivery Workers → Push Providers
• Redis for fast read & subscription cache; NoSQL for durable subscriptions & historical data; S3/object store for raw events.

Key notes

• Use topic exchange + shard mapping for RabbitMQ: shard = hash(city_id) % N_SHARDS.
• Redis keys:
  • forecast: weather::city::<city_id>::date::<YYYY-MM-DD>
  • subscribers: subscribers::city::<city_id>::shard::<n>
• Long-term storage: S3 (raw and canonical), Time-series DB (Timescale/ClickHouse) for analytics/ML.

4) Detailed component deep dives (3 components)

A. Subscription Service

• Purpose: persist subscriptions and serve fast per-city subscriber lists.
• How it works (steps): write POST /v1/subscribe/{city} → store in NoSQL (PK=city#<id>, SK=user#<id>) → invalidate/refresh Redis shard cache subscribers::city::<id>::shard::<n>.
• Data model: NoSQL row = {city_id, user_id, push_token, platform, notify_level, channels, status, updated_at}.
• Scaling: partition by city_id; hot-city sharding city#id#shard-<0..k-1>; autoscale DB throughput; Redis cluster for cached shards.
• Capacity (example): avg subs/city = Users/Cities → 1,000,000/10,000 = 100; hot-city (200k) → increase k until shard size ≈ target chunk (e.g., ≤ 10k entries).
• Algorithms / DS: Redis sets/lists for shard; consistent hashing for shard mapping; chunked iteration for large shards.
• Tradeoffs / failure modes: AP NoSQL for availability; eventual consistency between DB and Redis; mitigate with short TTLs and immediate invalidation on writes.

B. Kafka (topics) + Delivery Workers

• Purpose: durable stream of canonical events (normal/urgent) and scalable fan-out to workers for delivery.
• How it works (steps): providers → Kafka topics (weather.normal, weather.urgent) → Redis sink consumer updates cache → delivery consumer reads topic partitions → lookup subscribers → batch & send pushes → commit offsets.
• Topic/partitioning: partition by hash(city_id) % P; create separate urgent topic to prioritize processing.
• Scaling: increase partitions P for throughput; consumer group size ≤ P; autoscale workers by consumer lag and delivery latency.
• Capacity (example): events/sec ≈ (Cities × updates/hour)/3600 → 10,000×4/3600 ≈ 11.11; deliveries/sec ≈ events/sec × avg_subs → 11.11×100 ≈ 1,111. worker_count = ceil(deliveries/sec / worker_throughput) (e.g., throughput=400 → 3 workers baseline).
• Algorithms / DS: consistent hashing for partition assignment; batching per provider; idempotency keys delivered::<event_id>::<user_id> in Redis; DLQ on permanent failures.
• Tradeoffs / failure modes: Kafka gives durability and replay (no native priority — use separate urgent topic); ensure idempotent sinks and careful offset management to avoid duplicates.

C. Redis (cache + sink + dedupe)

• Purpose: low-latency reads for GET, fast subscriber cache for delivery, dedupe store for events/deliveries.
• How it works (keys):
  • forecast: weather::city::<id>::date::<YYYY-MM-DD> → JSON/value with TTL.
  • subscribers: subscribers::city::<id>::shard::<n> → serialized array or Redis set.
  • dedupe: event_dedup::<event_hash> and delivered::<event_id>::<user_id> → TTLed keys.
• Scaling: Redis Cluster with hash slots; shard hot keys across nodes; use memory eviction policies (volatile-lru) and separate clusters for cache vs dedupe if needed.
• Capacity (example): forecast key ~1 KB × 10,000 cities × 7 days = ~70 MB; subscriber shard memory depends on shard sizing (e.g., 200k subs × 200 bytes ≈ 40 MB).
• Algorithms / DS: sets/hashes for membership and lookup; Bloom filter for pre-filter fuzzy dedupe; pipelining for batch writes/reads.
• Tradeoffs / failure modes: in-memory speed vs persistence; on Redis failure fallback to NoSQL (slower) and use consumer replay to rebuild cache.