Requirements

Functional

• User-selected location: Users can request weather details for a chosen location (temperature, humidity, chance of rain, sunny/clear, etc.).
• Immediate crisis notifications: Users must be notified immediately when a crisis affects their subscribed location (storm, hurricane, high winds, flash flood, etc.).

Non-functional

• Availability & Partition Tolerance: System must prioritize availability and partition tolerance (AP in CAP). Small staleness (seconds) is acceptable.
• Scale target: Support ~1,000,000 daily active users (estimate → ~12 requests/sec steady average for read APIs). System must scale for bursts (e.g., city-wide crisis).

API Design

All endpoints are JSON over HTTPS. Inputs validated; errors use standard HTTP codes and a structured error body:

POST /v1/subscribe/{city}

Purpose: Subscribe (or update) a user for city notifications.

Path params

• city — canonical city identifier or slug (server resolves name → canonical id).

Request body

Responses

• 201 Created — subscription created (returns subscription resource).
• 200 OK — subscription updated.
• 400 Bad Request — validation error.
• 404 Not Found — unknown city.
• 409 Conflict — duplicate when required by semantics.

Notes

• Idempotent: repeat request with same user_id + city is safe (updates preferences).
• On success the subscription is cached in Redis for fast lookup.

DELETE /v1/unsubscribe/{city}

Purpose: Remove user subscription.

Request body (or auth context)

Responses

• 200 OK — unsubscribed.
• 404 Not Found — subscription not found.

GET /v1/weather/{city}/{YYYY-MM-DD}

Purpose: Return forecast for a city on a specific date.

Response (200 OK)

Errors

• 404 Not Found — city or date unknown
• 400 Bad Request — invalid date format
• 500 Internal Server Error — server failure

Notes

• Primary read path served from Redis cache (cache-aside). If cache miss, read canonical store, populate cache, return.

Edge Cases & Handling

1. Conflicting data from multiple providers

• Numeric fields (temp, humidity, wind): compute a weighted average using provider confidence/quality weights.
• Categorical/hazard fields: use max-severity and max-confidence rules (if any provider flags severe hazard → mark as urgent).
• Keep provider weights in config and update through ops without redeploy.

2. Duplicate or repeated events

• Each incoming canonical message carries event_id. Use Redis SET / bloom filter with TTL to dedupe within a short window.

3. Multiple alerts for same region

• Coalesce alerts by city_id + hazard_type + time_window (e.g., 15 minutes) to avoid notification storms; escalate if severity increases.

4. Missed deliveries (offline devices)

• Maintain a recent-event buffer per user (Redis list; 24h TTL). On reconnection, app can pull missed events or server retries according to user preference.

5. Hot-city (skew) handling

• For very large subscriber counts, shard subscriber partitions and fan-out across multiple delivery queues/workers. Batch deliveries to push providers and respect rate limits.

6. Provider downtime or degraded quality

• Fall back to last known forecast in Redis plus a “stale” flag. Notify ops and optionally inform users that data is stale.

7. Invalid user tokens / permanent push failures

• Mark tokens invalid after N permanent failures and surface them for cleanup; optional retry and user notification on next app open.

High-Level Architecture (concise)

• Ingestion / Forecast Gateway: receives provider feeds → normalizes → hazard detection → publishes canonical events.
• RabbitMQ (topic exchange + priority queues): routes events to sharded delivery queues; urgent events prioritized.
• Subscription Service & Datastore: stores per-city subscription records (partitioned by city_id) + Redis cache for fast lookup.
• Delivery Workers: consume messages, fetch subscribers, batch and send notifications to push providers (Airship/FCM/APNs).
• Redis: cache for forecasts (weather::city::<id>::date::<YYYY-MM-DD>) and cached subscriber sets.
• Long-term store: raw provider payloads and canonical events (S3/object store) for replay/audit.
• Monitoring & DLQ: metrics, tracing, DLQs for failed deliveries.

Detailed Component Design (deep dive on 3 key components)

A — Ingestion & Normalization Service (Deep Dive)

Purpose: Turn heterogeneous provider inputs into a canonical, enriched event and identify hazards.

Core responsibilities

1. Accept provider data (webhooks, pull, S3 drops).
2. Map provider schema → canonical schema (city_id, date, metrics, source, confidence).
3. Run hazard detection rules; tag urgency (LOW | NORMAL | HIGH).
4. Deduplicate events (Redis hash / SET with TTL).
5. Publish canonical message to RabbitMQ and update Redis forecast cache.
6. Persist raw + canonical payloads to S3 for replay/audit.

Scaling

• Stateless connectors deployed on K8s; autoscale by input rate.
• Backpressure via internal queue + RabbitMQ; throttle providers if necessary.

Algorithms / Data structures

• Dedup: Redis SET of event_id (TTL 5–15m) or content-hash bloom filter for fuzzy duplicate detection.
• Hazard rules: JSON-configurable rule engine; for complex tracking, add a small state store (time-series DB) to track geo-time sequences.
• Normalization: per-provider mapping tables and transformation pipeline (functional, idempotent).

Tradeoffs

• Stateless rules are simple and fast but miss multi-event patterns (storm tracking). Adding stateful tracking increases complexity and storage needs.

B — Subscription Service & Datastore (Deep Dive)

Purpose: Store and serve subscriptions for fast fan-out.

Data model (NoSQL recommended)

• Partition key: city#<city_id>
• Sort key: user#<user_id>
• Attributes: push_token, platform, notify_level, channels, status, created_at, last_invalidated_at

Why NoSQL (DynamoDB/Cassandra)

• Easy horizontal scale and high availability.
• Fast partitioned reads for city_id → subscribers fan-out pattern.
• Eventual consistency acceptable for low-latency notifications.

Caching

• Redis key: subscribers::city::<city_id>::shard::<n> storing serialized arrays/sets of subscriber objects (TTL 1–5 minutes).
• On subscribe/unsubscribe, update DB and invalidate/refresh cache.

Scaling & sharding

• Default partition by city_id. For hot cities, use shard suffix: city#<id>#shard-<0..k-1> where k grows with subscribers.
• Use consistent hashing to map city_id → shard number for delivery queue routing.

Algorithms / data structures

• Subscriber lookup: Redis set/list → O(1) membership reads.
• For very large subscriber sets, use chunked retrieval + concurrent worker batches.

Tradeoffs

• NoSQL: scales easily, but complex multi-row transactions are harder. If strict consistency on subscription updates is required, add a per-user confirmation or synchronous check on critical alert delivery.

C — Messaging & Delivery (RabbitMQ + Workers) (Deep Dive)

Purpose: Reliable, prioritized fan-out of events to subscribers.

Exchange & queue model

• Topic exchange with routing keys:
  • weather.city.<city_id>.normal
  • weather.city.<city_id>.urgent
• Sharded queues: deliver.shard.<n>.normal and deliver.shard.<n>.urgent
  • shard = hash(city_id) % N_SHARDS
  • Bind routing patterns to appropriate shard queues.

Priority & ordering

• Urgent queues processed before normal ones; urgent messages can use RabbitMQ priority or be placed in separate urgent queues for faster consumers.

Delivery worker steps

1. Consume message.
2. Resolve city_id → shard then fetch subscribers from Redis (or DB fallback).
3. Filter by notify_level and user preferences.
4. Batch notifications per push provider.
5. Send to push provider; on transient failures, retry with exponential backoff; on permanent failures, mark token invalid and write to DLQ.
6. Record delivery attempts in logs/analytics store.

Scaling

• Increase N_SHARDS and worker count to raise throughput. Autoscale worker pool based on queue length and delivery latency.
• For extreme bursts, autoscale aggressively and backpressure by controlling prefetch.

Algorithms / data structures

• Consistent hashing to map cities → shards (stable on shard count changes).
• Batching: group tokens into Provider-friendly batch sizes.
• Deduplication per user-event: Redis key delivered::<event_id>::<user_id> with TTL.

Capacity example (simple)

• If canonical events ≈ 12/sec and average subscribers per event = 100 → ~1,200 deliveries/sec. Batching and concurrency shape worker count; plan ~3–10 worker instances initially and autoscale by traffic.

Tradeoffs

• RabbitMQ gives per-message priority and routing flexibility, but operational complexity grows with sharding and mirrored queues. If throughput needs rise dramatically, consider Kafka for high throughput and replayability, but emulate priority via separate topics.

D — Redis Cache (Summary)

Purpose: Low-latency reads for forecasts and subscriber caches.

Keys

• Forecast: weather::city::<city_id>::date::<YYYY-MM-DD>
• Subscribers: subscribers::city::<city_id>::shard::<n>

Strategy

• Cache-aside for GET API. Writes from ingestion update Redis (write-through) to keep reads fresh.
• TTLs (5–30 minutes for forecasts; 1–5 minutes for subscriber lists). Use eviction and consistent hashing across Redis cluster nodes.

Tradeoffs

• Fast, but memory-limited. Hot keys must be sharded; cache misses incur DB latency.

E — Push Providers & Delivery Service (Summary)

Purpose: Interface with Airship / FCM / APNs (or custom push gateway) for final delivery.

Considerations

• Use provider batch APIs where available.
• Respect rate limits and backoff policies.
• Provide provider-agnostic abstraction so new providers can be added with minimal change.
• Track success/failure metrics and retry/cleanup invalid tokens.

Flow Summary (one-line steps)

1. Subscribe: User → POST /v1/subscribe/{city} → Subscription DB + Redis cache.
2. Ingest: Provider → Ingestion service → normalize → hazard detect → publish to RabbitMQ + write forecast to Redis.
3. Route: RabbitMQ topic exchange routes to sharded queues (urgent vs normal).
4. Deliver: Workers consume, fetch subscribers, batch, and send via push providers (retries + DLQ).
5. Read: GET /v1/weather/{city}/{date} → Redis (cache-aside) → fallback to canonical store if needed.