Design a distributed Notification System

Question

Design a distributed notification system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Users can create, update, and delete notifications.
  - Users can subscribe to specific types of notifications (e.g., comments, mentions).
  - Notifications should be delivered in real-time (or near real-time).
  - Users can view a history of their notifications.
  - Support for push notifications on mobile and web platforms.

- **Non-Functional Requirements**:
  - **Scalability**: The system must handle millions of notifications per second, especially during peak times.
  - **Availability**: The system should be available 99.9% of the time.
  - **Latency**: Notifications must be delivered to users within a few seconds.
  - **Durability**: Once a notification is sent, it should persist even in the event of a failure.

- **User Base**: Assume Notion has 10 million active users.
- **Notification Rate**: If each user generates an average of 5 notifications per day:
  - Total notifications per day = 10 million users * 5 notifications = 50 million notifications.
  - Average QPS (queries per second) = 50 million notifications / 86400 seconds = ~578 QPS.
- **Storage Needs**: If each notification is approximately 1 KB, then:
  - Daily storage = 50 million notifications * 1 KB = ~50 GB/day.
  - Monthly storage = ~1.5 TB/month. Considering a 30-day retention policy, we would need around 45 TB for storage.
- **Bandwidth**: Assuming notifications are sent over a network with an average payload of 1 KB, total bandwidth required:
  - 50 million notifications/day * 1 KB = ~50 GB/day.

The architecture consists of the following components:
- **API Gateway**: Handles incoming requests, rate limiting, and authentication. Technologies: NGINX or AWS API Gateway.
- **Notification Service**: Responsible for creating, updating, and deleting notifications. Technology: Node.js with Express.
- **Message Queue**: For real-time processing and decoupling of services. Technology: Apache Kafka.
- **Database**: Stores notifications and user preferences. Technology: PostgreSQL for structured data and Redis for caching.
- **Push Notification Service**: Sends notifications to devices. Technology: Firebase Cloud Messaging (FCM).
- **Monitoring and Logging**: Tools like Prometheus for monitoring and ELK stack for logging.

Schema design for notifications:
- **Users Table**:
  - user_id (PK, UUID)
  - username (String)
  - preferences (JSON)

- **Notifications Table**:
  - notification_id (PK, UUID)
  - user_id (FK, UUID)
  - type (String)
  - message (String)
  - timestamp (TIMESTAMP)
  - read_status (Boolean)

- **Indexes**:
  - Index on user_id for fast retrieval of notifications for a user.
  - Index on timestamp for querying recent notifications.

This schema allows efficient querying and supports user preferences for filtering notification types.

- **Consistency vs. Availability**: Given the requirement for high availability, we can opt for eventual consistency in notification delivery. This means notifications may not be delivered in real-time but will eventually reach the user, which is acceptable in this context.
- **Database Choices**: Choosing PostgreSQL for structured data gives us strong consistency guarantees, but we may face scalability issues with high write loads. Redis is used as a cache to mitigate read latencies for frequently accessed notifications.
- **Partitioning Strategy**: We can partition the notifications table by user_id to distribute the load evenly across database nodes. This helps in scaling write operations and improves read performance.
- **Failure Handling**: Implementing retries for failed message deliveries and ensuring idempotency in the Notification Service helps handle transient failures efficiently.
- **Monitoring and Alerting**: Setting up alerts for high error rates and monitoring system performance metrics (like QPS, latency) will help maintain system health.

Design a distributed Notification System

Notion

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Notion Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search