Detailed Component Design (deep dive on 3 key components)

A — Ingestion & Normalization Service (Deep Dive)

Purpose: Turn heterogeneous provider inputs into a canonical, enriched event and identify hazards.

Core responsibilities

Accept provider data (webhooks, pull, S3 drops).
Map provider schema → canonical schema (city_id, date, metrics, source, confidence).
Run hazard detection rules; tag urgency (LOW | NORMAL | HIGH).
Deduplicate events (Redis hash / SET with TTL).
Publish canonical message to RabbitMQ and update Redis forecast cache.
Persist raw + canonical payloads to S3 for replay/audit.

Scaling

Stateless connectors deployed on K8s; autoscale by input rate.
Backpressure via internal queue + RabbitMQ; throttle providers if necessary.

Algorithms / Data structures

Dedup: Redis SET of event_id (TTL 5–15m) or content-hash bloom filter for fuzzy duplicate detection.
Hazard rules: JSON-configurable rule engine; for complex tracking, add a small state store (time-series DB) to track geo-time sequences.
Normalization: per-provider mapping tables and transformation pipeline (functional, idempotent).

Tradeoffs

Stateless rules are simple and fast but miss multi-event patterns (storm tracking). Adding stateful tracking increases complexity and storage needs.

B — Subscription Service & Datastore (Deep Dive)

Purpose: Store and serve subscriptions for fast fan-out.

Data model (NoSQL recommended)

Partition key: city#<city_id>
Sort key: user#<user_id>
Attributes: push_token, platform, notify_level, channels, status, created_at, last_invalidated_at

Why NoSQL (DynamoDB/Cassandra)

Easy horizontal scale and high availability.
Fast partitioned reads for city_id → subscribers fan-out pattern.
Eventual consistency acceptable for low-latency notifications.

Caching

Redis key: subscribers::city::<city_id>::shard::<n> storing serialized arrays/sets of subscriber objects (TTL 1–5 minutes).
On subscribe/unsubscribe, update DB and invalidate/refresh cache.

Scaling & sharding

Default partition by city_id. For hot cities, use shard suffix: city#<id>#shard-<0..k-1> where k grows with subscribers.
Use consistent hashing to map city_id → shard number for delivery queue routing.

Algorithms / data structures

Subscriber lookup: Redis set/list → O(1) membership reads.
For very large subscriber sets, use chunked retrieval + concurrent worker batches.

Tradeoffs

NoSQL: scales easily, but complex multi-row transactions are harder. If strict consistency on subscription updates is required, add a per-user confirmation or synchronous check on critical alert delivery.

C — Messaging & Delivery (RabbitMQ + Workers) (Deep Dive)

Purpose: Reliable, prioritized fan-out of events to subscribers.

Exchange & queue model

Topic exchange with routing keys:
- weather.city.<city_id>.normal
- weather.city.<city_id>.urgent
Sharded queues: deliver.shard.<n>.normal and deliver.shard.<n>.urgent
- shard = hash(city_id) % N_SHARDS
- Bind routing patterns to appropriate shard queues.

Priority & ordering

Urgent queues processed before normal ones; urgent messages can use RabbitMQ priority or be placed in separate urgent queues for faster consumers.

Delivery worker steps

Consume message.
Resolve city_id → shard then fetch subscribers from Redis (or DB fallback).
Filter by notify_level and user preferences.
Batch notifications per push provider.
Send to push provider; on transient failures, retry with exponential backoff; on permanent failures, mark token invalid and write to DLQ.
Record delivery attempts in logs/analytics store.

Scaling

Increase N_SHARDS and worker count to raise throughput. Autoscale worker pool based on queue length and delivery latency.
For extreme bursts, autoscale aggressively and backpressure by controlling prefetch.

Algorithms / data structures

Consistent hashing to map cities → shards (stable on shard count changes).
Batching: group tokens into Provider-friendly batch sizes.
Deduplication per user-event: Redis key delivered::<event_id>::<user_id> with TTL.

Capacity example (simple)

If canonical events ≈ 12/sec and average subscribers per event = 100 → ~1,200 deliveries/sec. Batching and concurrency shape worker count; plan ~3–10 worker instances initially and autoscale by traffic.

Tradeoffs

RabbitMQ gives per-message priority and routing flexibility, but operational complexity grows with sharding and mirrored queues. If throughput needs rise dramatically, consider Kafka for high throughput and replayability, but emulate priority via separate topics.

D — Redis Cache (Summary)

Purpose: Low-latency reads for forecasts and subscriber caches.

Keys

Forecast: weather::city::<city_id>::date::<YYYY-MM-DD>
Subscribers: subscribers::city::<city_id>::shard::<n>

Strategy

Cache-aside for GET API. Writes from ingestion update Redis (write-through) to keep reads fresh.
TTLs (5–30 minutes for forecasts; 1–5 minutes for subscriber lists). Use eviction and consistent hashing across Redis cluster nodes.

Tradeoffs

Fast, but memory-limited. Hot keys must be sharded; cache misses incur DB latency.

E — Push Providers & Delivery Service (Summary)

Purpose: Interface with Airship / FCM / APNs (or custom push gateway) for final delivery.

Considerations

Use provider batch APIs where available.
Respect rate limits and backoff policies.
Provide provider-agnostic abstraction so new providers can be added with minimal change.
Track success/failure metrics and retry/cleanup invalid tokens.

Flow Summary (one-line steps)

Subscribe: User → POST /v1/subscribe/{city} → Subscription DB + Redis cache.
Ingest: Provider → Ingestion service → normalize → hazard detect → publish to RabbitMQ + write forecast to Redis.
Route: RabbitMQ topic exchange routes to sharded queues (urgent vs normal).
Deliver: Workers consume, fetch subscribers, batch, and send via push providers (retries + DLQ).
Read: GET /v1/weather/{city}/{date} → Redis (cache-aside) → fallback to canonical store if needed.

Submissions

Displaying Solutions with Scores Ranging from 1 to 10

Submissions

Displaying Solutions with Scores Ranging from 1 to 10