Design a distributed Notification System

Question

Design a distributed notification system handling millions of requests. Discuss consistency, availability, and performance trade-offs.

Codemia · Accepted Answer

- **Functional Requirements:**  
  - Users can subscribe/unsubscribe to notifications based on preferences (e.g., promotions, order updates).  
  - Notifications can be sent via multiple channels (e.g., email, SMS, app push notifications).  
  - Admins can create and schedule notifications for specific events (e.g., sales events, restock alerts).  
  - Users can view their notification history in the app.

- **Non-Functional Requirements:**  
  - **Scalability:** Must handle millions of notifications per minute, with the ability to scale horizontally.  
  - **Availability:** The system should be highly available, aiming for 99.9% uptime.  
  - **Consistency:** Eventual consistency is acceptable for notifications.  
  - **Latency:** Notifications should be delivered in real-time or near real-time (within seconds).  
  - **Security:** User data must be protected, with secure channels for communication.

Assuming Walmart has 100 million active users, and each user receives on average 10 notifications per week:  
- Total notifications per week = 100,000,000 users * 10 notifications = 1 billion notifications.  
- Daily notifications = 1 billion / 7 ≈ 143 million notifications.  
- Peak hour estimation (assuming 10% of daily notifications are sent in peak hour) = 14.3 million notifications per hour ≈ 4,000 notifications per second.  
- Considering a 3x overhead for retries and failed deliveries, we need to support around 12,000 notifications per second.

The high-level architecture of the notification system consists of several components:  
- **API Gateway:** Exposes RESTful endpoints for users and admins to send and manage notifications.  
- **Notification Service:** Responsible for processing notification requests and interfacing with message queues.  
- **Message Queue (e.g., Apache Kafka):** Handles high-throughput asynchronous processing of notifications.  
- **Notification Channels:** Multiple microservices for different delivery methods (e.g., Email Service, SMS Service, Push Notification Service).  
- **Database (e.g., Amazon DynamoDB):** Stores user preferences, subscription statuses, and notification history.  
- **Monitoring and Logging:** Tools like Prometheus and ELK Stack for observability and metrics collection.

This architecture supports scalability and fault tolerance.

The schema for the notification system includes:  
- **Users Table:**  
  - `user_id` (PK)  
  - `email`  
  - `phone_number`  
  - `preferences` (JSON)

- **Notifications Table:**  
  - `notification_id` (PK)  
  - `user_id` (FK)  
  - `type` (Enum: Promotion, Order Update, etc.)  
  - `content`  
  - `status` (Enum: Sent, Failed)  
  - `timestamp`

- **Subscriptions Table:**  
  - `subscription_id` (PK)  
  - `user_id` (FK)  
  - `channel` (Enum: Email, SMS, Push)  
  - `is_active` (Boolean)

This design allows efficient querying of user preferences and notification history.

- **Consistency vs. Availability:**  
  - Opted for eventual consistency to ensure high availability and performance, allowing notifications to be processed asynchronously.

- **Complexity vs. Performance:**  
  - Using microservices for notification channels introduces complexity but allows scaling of individual components based on load.

- **Delivery Guarantees:**  
  - Chose at-least-once delivery for notifications, which may result in duplicates but ensures that users receive important messages.  
  - Implemented deduplication logic in the Notification Service to handle potential duplicates effectively.

- **Cost vs. Performance:**  
  - Selected managed services like DynamoDB and Kafka for ease of scaling and maintenance, accepting higher costs for operational efficiency.

Design a distributed Notification System

Walmart

How to Approach This

Sharpen Your Skills on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Walmart Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search