Requirements

Functional Requirements:

1. Notifications can be sent to a single user or a user group, and can also be targeted to device/browser.
2. Support different notification types (push notification to iOS and Android, SMS message, and email)
3. Enforce user preferences, opt-outs, do-not-disturb windows, per-channel limits, before queuing any push
4. Can be triggered by users or by the server, or provide scheduling so a message can be queued for any future timestamp

Non-Functional Requirements:

1. Notification should be real-time (pushed to the user within one minute for 99% of the time)
2. Should ensure 99.9% of successful delivery with retry mechanism
3. Should scale up to support 50M DAU and 200M notifications per day
4. Should have high durability and ensure no data is lost even during crash/outage

API Design

Client trigger/schedule notification:

POST /notification/create {

"user_id": "123",

"timestamp": datetime,

"type": "PUSH" | "SMS" | "EMAIL",

"query": {}, including parameters for querying notifications from the server

"push_timetsamp": datetime (optional)

} -> create notification object in the database, with initial status.

Client create/update notification settings:

POST/PATCH /notification/settings {

"user_id": "123",

"timestamp": datetime,

"settings": {},

}

Server push notifications to the client (via WebSocket):

POST /notification/push {

"id": "456",

"user_id": "123",

"timestamp": datetime,

} -> push the notification to the client with updated contents (if delivery is confirmed, update status in the database)

High-Level Design

Client:

1. Send requests for querying notifications in real-time or schedule ahead
2. Receive notifications from the server from different sources, for example, APNs(Apple Push Notification service) for iOS, FCM (Firebase Cloud Messaging) for Android, and third-party services for SMS and email.

Notification server:

1. Process and parse requests from the client OR generate notification from the server side (promotion, service update, etc.), create notification object, and insert into the database
2. Append notification id to the message queue

Message queue:

1. Maintain notifications to be sent to users, and decouple the notification trigger/push process
2. Partition by user id, and keep chronological order for the same user
3. Can be configured to categorize notifications into high, medium, and low priority, impacting their processing order.

Push server:

1. Process notification from the message queue and route to the corresponding user via WebSocket
2. Update notification status as success after successful delivery or failed after timeout

Redis cache:

1. Cache frequently accessed data, including user group list, user device list, and notification templates, etc.
2. Use a distributed lock for every notification id, if the delivery is pending or successful, lock the notification to avoid duplicate delivery. If the delivery fails, release the lock for retry.

Database:

1. Use notification table to manage notification metadata and push status
2. Use user table to manage user preferences

Notification {

"id": str, - primary key

"user_id": str, - index

"content": {},

"create_time": datetime,

"schedule": datetime, - optional

"status": "UNPUSHED" | "SUCCEED" | "FAILED"

}

User {

"id": str, - primary key

"setting": {}, allow or opt-out for any type of notification, and also rate-limits

}

Detailed Component Design

Scalability:

1. Horizontal scale and shard database/message with user id
2. Replicate the cache and database to avoid single point failure, with write operations going to the primary node, and propagate to follower nodes, while read operations can be handled by all nodes.

Latency:

1. After the user triggered a notification, return to the user immediately without confirmation, while the notification is queued and waiting to be processed asynchronously.
2. If any user device is inactive, the notification delivery can not be completed. We can use another message queue for storing notifications for inactive devices. When devices come back online, they can query the queue to see if any pending notifications are waiting to be delivered. If a device remains offline for an extended period, the system can flag it for cleanup. This helps in managing the overall system load by removing stale devices.

Reliability:

1. Implement retry mechanism with exponential backoff for the notification message queue(also take into account the type and cause of failure), and reach a final failure if it exceeds the max retry count or timeout (move to another message queue for fault handling or review)
2. Immediate Validation on Delivery Attempts: When a notification is sent to a device, check the acknowledgment from the device. If the push notification service (using WebSockets or similar) receives a response indicating the token is invalid or the device is unreachable, mark the token as invalid immediately. And this device will be removed from the push list, unless users update their settings preference.