System requirements

Functional:

support different apps in mobile application

performance monitoring

message targeting: user preference, based on user behavior, user segment(location, demographics, etc)

delivery schedule: timezone, frequency, after some event

personalization notification

localization

user engagement tracking

Non-Functional:

reusability

multi-tenancy

high performance

reliability

scalability

Capacity estimation

known metrics:

100K notifications per second in peak, 115 in normal

1s e2e latency

5s of scheduled time

delivery rate 95%

notification open rate 20%~30%

error rate 2%

meesage payload length avg: 2k

QPS:

peak

100K, considering each instance send 10K, need 10 worker instance

non peak

115, need 1 worker

-- need auto scaling policy, talk later

networkband width:

peak 100K * 2K = 200M

non peak

115 * 2K = 200K

disk:

1 year of storage

115 * 24 * 3600 * 365 * 2K = 7253G = 7T storage

replica = 3, total 21T

API design

/send_notification, title, payload, receiver_id, device_type

return OK/NOT OK

/notification/history?receiver_id

return [notifcation_id, notification_payload, timestamp]

/user/preferences?receiver_id

return [recommend notification_id]

Database design

noSQL db

key=payload_id

value = payload

1. User Preferences Table

• user_id (Primary Key)
• preferences (JSON)
• timezone
• last_active

2. Notification Schedule Table

• notification_id (Primary Key)
• user_id
• message
• delivery_time
• status

3. Notification Logs Table

• log_id (Primary Key)
• notification_id
• status
• error_message

kafka

seperate topic for each destination(APn, fcm, http)

message_id = payload_id

value = payload

Database: PostgreSQL for user data and schedules; Redis or DynamoDB for fast lookups and caching.

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

Request flows

Message Creation and Personalization:

• A new notification request arrives at the API Gateway.
• The Worker fetches user data and applies personalization (e.g., user behavior, preferences, location).

Scheduling and Queuing:

• The Scheduler checks the delivery time and queues the message for delivery.
• Messages are grouped by time zone and event triggers for efficient delivery.

Notification Delivery:

• Worker instances pull from the queue and send notifications via appropriate channels (e.g., push, SMS, email).
• Delivery confirmation and error handling are managed at the worker level.

Monitoring and Feedback:

• Monitoring System collects metrics on delivery success, latency, and user engagement.
• Analytics provides insights into user behavior and notification performance.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

A/B testing

GDPR when storing PII

user activity reporting loop

API Gateway

• Handles authentication, rate limiting, and request validation.
• Scalable using load balancers and managed services (e.g., AWS API Gateway).

Scheduler

• Periodic job scanning the schedule table to pick messages ready for delivery.
• Uses distributed task queues (e.g., Redis, Kafka).

Worker Pool

• Horizontal scaling to handle peak traffic (100K QPS).
• Auto-scaling based on CPU utilization and queue depth.

Notification Delivery Service

• Integrates with third-party push notification services like APNs (Apple) and FCM (Firebase).

Monitoring & Analytics

• Uses Prometheus/Grafana for performance metrics.
• Logs user engagement (open rate, clicks).

Trade offs/Tech choices

Database: Use PostgreSQL for relational data (schedules) and Redis for fast access to real-time preferences.

Queue System: Kafka for high-throughput, fault-tolerant message processing.

Worker Scaling: Kubernetes-based auto-scaling ensures dynamic capacity.

Personalization & Localization: Preprocess personalized content to avoid runtime delays.

Delivery Guarantees: Use exponential backoff for retries to manage failure scenarios.

Failure scenarios/bottlenecks

Database Bottleneck: Redis caching can reduce read pressure on the primary database.

Queue Overload: Rate limiting at the API level prevents message spikes.

Notification Delivery Failure: Implement retry logic and fallback mechanisms.

Scheduler Delay: Optimize with sharded scheduling to parallelize message dispatch.

Future improvements

auto scaling for peak/non peak

machine learning process: separate recall and ranking

separate FCM workers to APN workers

backend server send to influx

have a calibration between backend servers to monitoring any data loss/duplication

Machine Learning for Targeting: Improve message targeting based on user behavior patterns.

Multilingual Support: Expand localization to handle more languages.

Real-time Analytics: Use streaming solutions like Apache Flink for near-real-time user engagement tracking.

A/B Testing: Experiment with different message formats to improve open rates.