Requirements

Functional Requirements:

• System should support sending notificaitons through Push, Email and SMS
• User should be able to control how they want to recive notificaitons - the service uses user preferneces to send notification
• System should be manage device token lifecycle
• System should support bulk and scheduled notificaitons
• System should optinally provide a way to check notification status

Non-Functional Requirements:

• Highly Scalable - system should be scalable to support millions of notificaitons per day
• Reliable and durable - no notifications that enters the service should be lost.
• Highly Available - Service should be highly available as many internal services maybe depended on it ~99.9% uptime
• Noticiatons are idempotent - same notificatiosn should not be sent more than once.

Capacity Estimation

• 500M notifications across all methods
• ~300M - push, 100M email, 100M - sms
• Pusk: 3k/sec, email/sms - 1k/sec
• Peak: 5x -> Push: 15K/sec, email/sms - 5k/sec

API Design

Notifications

POST /notifications

req :

{

userId,

template{

type,

verison

}

variables[],

scheduledTime?,

idempotentKey

}

POST /notifications/bulk - max 100 per batch

req:

{

users[],

template{

type,

verison

}

variables[],

scheduledTime?

}

GET /notifications/{id}/status

User Device token

POST /users/{id}/device - create device and add token

PUT /users/{id}/device/{id} - use for updating device status/remove device

PUT /device/{id}/token - update token

High-Level Design

API Gateway

Acts as the entry point for all client requests.

Responsibilities:

• Authentication (if required)
• Rate limiting - Applied using token bucket per IP or API
• Request routing

Load Balancer

Distributes traffic across multiple instances of all services to improve scalability and availability.

Notification Service

Responsibilities

• Recieves send notificaion request
• Validates the request - Auth, user valdiation, notification type etc
• Checks all users preference for channels of the notificaiton type
• Creates Notifcation Request record and notification record
• Sets the status to Accepted and Pending respectively
• Fans out the request to all subscrived channels to its own queue.
• uses idempotent key to ensure same request that coems again is not processed again.

Email/SMS/Push Queue

• Kafka backed - Acts as a durable event message que
• Uses userId as partition key - preservers ordering/processing of notifications

Email/SMS worker

• Consumes messages from respected queue and processes them
• Updates notification request status to processing
• Creates notification body using template DB
• Sends out the request to external provider - twilio
• Updates notification status to Completed after receiveing response back,
• We use External provider Idempotency key incase if external provider completed request but our DB failed to update status- this ensures when request is processed again, the External provider is sent the same key and its returns back the respionse it had when it rpocessed earlier.

Push Worker

• Same responsiblities as Email/SMS worker regarding building noticiaiton ody and updating statutes
• Here we dont use external providers but use FCM and APNs SDKs
• So instead of idemptonet keys, we ensure same request does not process again by using the enrire transaction as Atomic transaction.

Scheduler

• Runs periodically and chesk notificaiton table to see if any noticiation are scheduled to go out,
• Picks those up and sends to its respected queue for processing

Device Service

• Used to manage device token lifecycle
• Acts as Device token CRUD

Devide DB/TempalteDb/NoticaitonDB/User PrefrecnesDB

• Persistent storage used for storing data

Database Design

I would use a hybrid approach. Notification delivery data is high-volume and mostly key-value access, so I would use NoSQL like Cassandra for user preferences, device tokens, notification requests, and notifications. For template managemen which is admin managed and where versioning and consistency matter, I would use SQL like PostreSQL.

UserPreferecnes

userId

notificaitonType

email

sms

push

Device

deviceId

userId

devideToken

status -> Active/InActive

NotificationRequest

Notification

Template

id

notificationType

content

channel

status

version

Detailed Component Design

now we will deep dive into this topics

How to ensure Notifications are not lost once notificaiton service publishes it.

Kafka acts as durable event broker giving at-least once delivery gurantee.

• Kafka persists messages
• So if a worker fails mid processing, the message is still in kafka
• WHne the worker is back upa again, it will pick up the message and process it, so no messages/noticcations are lost.
• Secondly, for some reason kafka also failed and message was not persist, then we already have the notificaiton record saved in DB
• Our scheduler can also look for notifications recoed that has been sitting as 'Pending'/'Processing' for a long time, and it can pick it up for processing

Managing Device Token Lifecycle

• This is very crucial for our Push notificaitons to work.
• Device service takes care of it.

Registering Device

• When a user isntalls app
• The app will send request to device service to register the device with a devide token
• Device serviec will cereate a record and store the device token

Updating TOken

• When a user reinstall the app or the mobile OS rotates the token
• The App will call device service to update the token
• Device service will fidn the record and update the token

How does Bulk and Scheduled Notifcaiton work

-> For bulk,

• We expose a bulk endpoint for clients to use
• max limit is 100 users per request
• Notification service will receive this request
• It will create one noticaiton request recoed with type Bulk
• And for each user it will create a notication record for all of their preferred channels
• now the main differnce, isnteasd of sending all those requests one by one to queue, we send only one to each queue
• We only send to queue, if we created a record for that channel, or else skip thjat entirely
• Each chanbel queue wiell recive the request with type 'BULK' and requsetId
• The worker will then pocess all notifications with the noticaitonRequestid
• The worker process for bulk/single could be the differentiate, For bulk we can implement parallel/concurrent processing.

-> The main differnce is our service is not respoinbile for fanning out to millions of user in just one request, for eg: A yotube channel posted a video -> notified all of its subsciber which could be millions,

-> Our serviec wont handle this mainly because, for that to happen, our service will then need to know the relationships/data of clients, so its better cleints coem up with strategies for batch procesisng and use our bulk endpoint for that.

For scheduled

• Two services come in play: Notificaiton + Scheduler
• Notifiation does all the things it does for normal flow accept sending or fan out to channel quues
• So notification just create records but wuith 'scheduledTime' filled
• Scheduler will then be responsbile to pick up the notifications when its due
• Scheduler most likely checks the DB each second and processes the due notificaitons.
• It will send the notificaiton to its respsected quueue
• From ther the processing is same

Failure Handling

Consumer/Worker failer ->

• do not commit kafka offset
• kafak retires the same message
• After x retires the message will moved to dlq

Duplicate processing

• Client can send same notification request twice because of some network glicth
• We use idemptoent key -> sent by client
• This ensures same request is not processed twice

• Kafka is at least once delivery guarantee
• So same message can come up
• We will ensure this dont happen by having a light validation on status -> consumer checks if status of the ntoication is not ocmpleted then only proceed

• Ther might be case where provider sent noticiaotn but updating staus failed
• This amyu result in sending noticiotna twice
• We mitigate this via using idempotent key on provider seide
• So each request to provider will also send idempotent key
• This ensures that when a rqeust is sent to provider for already delviered noticaiton, its not delvied again rather the staus is send back

Provider failure -> retries with expotential backoff

Scalability

The system scales horizontally by partitioning Kafka topics, using stateless channel workers, sharding high-volume notification storage by userId/time, and autoscaling workers based on queue depth.