System Requirements

Functional Requirements:

• Deliver asynchronous messages across multiple channels: iOS Push (APNs), Android Push (FCM), SMS (e.g., Twilio), and Email (e.g., SendGrid).
• Allow clients to trigger notifications via a unified REST API using templated payloads.
• Support user-level routing preferences and strict opt-outs.
• Track delivery statuses, hard bounces (invalid tokens/numbers), and user engagement (opens/clicks).

Non-Functional Requirements:

• Instantaneous API Response: The ingestion API must return a 202 Accepted immediately, decoupling the internal system from external provider latencies.
• High Availability & Fault Tolerance: The system must survive downstream provider outages (e.g., APNs going down) without losing messages.
• Scalability (Thundering Herd Protection): Must gracefully handle massive, sudden spikes in traffic (e.g., breaking news alerts) using buffered queues and rate limiting.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

1. Trigger a Notification POST /v1/notifications/send

JSON

Returns: 202 Accepted. The API validates the payload, drops it into the initial Kafka/RabbitMQ ingestion topic, and returns immediately.

High-Level Design

• API Gateway: Authenticates internal microservices calling the notification API.
• Ingestion Service: Validates the schema, checks idempotency keys, and publishes the raw request to the incoming_notifications message broker.
• Message Broker (Kafka / RabbitMQ): The highly durable buffer handling queues for routing, specific providers, and retries.
• Router & Enrichment Workers: The "brain" of the system. Consumers that pull from the incoming queue, check user preferences, fetch device tokens/phone numbers, render templates, and fan-out messages to provider-specific queues.
• Sender Workers: Isolated worker pools dedicated to specific external APIs (FCM, APNs, Twilio, SendGrid).
• Analytics / Webhook Ingestion: A service dedicated to receiving delivery receipts and bounce events from external providers to clean up dead tokens.
• Fast Cache (Redis): Handles rate limiting and 24-hour idempotency checking.

Detailed Component Design

Here are the refined mechanics of the resilient delivery pipeline.

1. Idempotency & Rate Limiting (Redis)

Before the Ingestion Service puts a message on the queue, it checks Redis for the notification_id. If it exists, the request is dropped as a duplicate. Next, it increments a rate-limiting counter for that user_id. If the user has received > 10 notifications in the last hour, the message is either dropped or shunted to a low-priority digest queue.

2. The Router & Fan-Out Process

Router Workers pull from incoming_notifications.

• They query the PostgreSQL User Preferences table. If the user has opted out of "Marketing SMS", that specific channel is dropped.
• They query the Device/Contact table to fetch the FCM token, APNs token, or Phone Number.
• They format the specific payload required by Apple, Google, or Twilio.
• Finally, they push the formatted, ready-to-send payload into isolated, provider-specific queues (e.g., fcm_queue, apns_queue, sms_queue).

3. Resilient Sending & Exponential Backoff

Sender Workers pull from their respective provider queues and execute the HTTP calls.

• Success (200 OK): An event is published to the analytics_queue.
• Hard Failure (400 Bad Request / 404 Unregistered): The token or phone number is invalid. An event is sent to the Analytics Service to immediately soft-delete that token from the database so we never query it again.
• Soft Failure (429 Rate Limited / 503 Service Unavailable): The worker pushes the message to a Retry Queue. Crucially, it applies Exponential Backoff with Jitter (e.g., wait 2s, then 5s, then 12s, plus a random millisecond offset) to ensure our retry workers don't accidentally DDoS the external provider when it comes back online.

Database Design

The data storage is split to handle configuration versus high-volume logging.

1. The Configuration Store: PostgreSQL (Users & Devices)

Designed for strong consistency to ensure strict adherence to user privacy and opt-outs.

2. The Delivery Log Store: Cassandra / MongoDB (Analytics)

Designed to absorb the massive write-heavy firehose of delivery receipts, opens, and clicks without slowing down the core sending pipeline.