Detailed Component Design

Now we delve deeper into some critical components: the Delivery Engine (workers and connection handling), the Scheduler (timing management), and the Retry mechanism for failures. These are complex parts that ensure the system is robust and meet the requirements for timely delivery and reliability.

Delivery Engine (Push Workers)

The Delivery Engine is the component responsible for actual transmission of notifications to user devices. In our design, it consists of multiple Delivery Worker processes, which may be specialized by platform or combined, and a supporting infrastructure for device connections.

Connection Management

Since we are not using platform push networks, our server must maintain a connection to each device to deliver notifications in real-time. A common choice is to use WebSocket connections (over TLS) for mobile and web clients. When the mobile app or web app starts, it opens a WebSocket to our push service (e.g., to wss://push.myapp.com). We can have a separate set of servers (connection gateway) handling the initial WebSocket upgrade and then assigning that connection to a Delivery Worker process. We need to track which user/device is on which connection. We could maintain an in-memory map, but since there could be millions, we might partition them. One approach: hash the device ID to one of N worker instances so that the device always connects to a specific machine (or cluster). In practice, we might use a load balancer that hashes by a session cookie or device ID to route to the same backend. Alternatively, a central registry (like a Redis or even our Device table) could store a mapping: device -> current connection server. To keep things speedy, each Delivery Worker process could store the connections it manages in memory and also register them in a distributed cache for global lookup.

Horizontal Scaling of Connection

If we have 10 million devices, and say we run 100 worker servers, that’s an average of 100k connections per server. This is plausible with modern networking (with async I/O, each server can handle even more). We ensure the OS is tuned (high file descriptor limits, and using non-blocking sockets). Using event-driven frameworks (like Node.js or Netty in Java or asyncio in Python or Go’s goroutines) is important to handle many simultaneous connections without spawning a thread per connection. Each connection idle overhead is small (a few KB of memory).

Message Routing to Connections

When a Delivery Worker receives a message from the queue (e.g., “notif 123 to device X”), it needs to find if device X is connected to this worker. If our routing of messages is aligned with connections, ideally the message is only consumed by the worker that holds device X’s connection. We can enforce that by using the device ID as a partition key in Kafka, so all notifications for device X go to the same partition/consumer.

We then assign consumers in such a way that the one with device X’s connection is the one consuming that partition. This is tricky to set up dynamically, so instead we might make the consumption of the queue stateless and do a lookup: any worker can get a message for any device, then it checks a central map “which server has device X?”. If it’s itself, great, send directly. If it’s another server, there are two options: (a) forward that message to the correct server (via an internal message bus or RPC), or (b) have only the correct server pull that message in the first place. The latter is more efficient. Kafka consumer grouping typically doesn’t ensure consumer affinity by data content out of the box; we might need a custom assignment strategy.

Alternatively, we separate the concerns: Instead of one global queue for devices, we could have each server maintain its own queue of messages for devices it owns. The router then would need to publish to specific server queues. This could be done if the router knew the device’s home server at fan-out time (looking it up in device registry). That might be feasible: the Device registry could have a field for “current_server_id” for each device that updates on connection. Then the router can direct message “notif 123 for device X” directly to server 7’s queue. This is akin to how some chat systems route messages to the server holding a user’s session. It simplifies delivery (no extra hop), at the cost of dynamic routing logic.

To keep design simpler, let’s assume our message broker approach can handle device-affinity (e.g., by partition key or by separate topics per server). If not, an intermediate “Connection Manager service” might be needed to forward to the right worker.

Sending the Notification

Once the worker identifies the connection for device X (which should be an open socket or similar), it serializes the notification into a message format. This could be JSON or a binary protocol. It then writes it to the socket. Because it’s TCP, this is reliable transport. If the device is online, it will receive the data almost instantly. We may implement an acknowledgment from the client, e.g., the app could respond “notif 123 received” which could be used to mark it as delivered on our side and possibly remove it from any retry queue. If we don’t get an ack within a short time, we might consider it failed (but TCP is pretty reliable; lack of ack likely means the connection broke).

Handling Failures in Send

If writing to the socket fails (throws an exception, or we realize the socket is closed), the worker knows the device is offline. It will update the device’s status (maybe mark in registry that it’s disconnected now with last_seen). It then triggers the retry flow: perhaps put the message back on a retry queue or mark it for later. If a connection is closed, the device will probably reconnect later; we can then deliver any pending messages.

Security Considerations

The connection is authenticated (likely the device sent a user token when connecting, so we associate that connection with a user ID). We must ensure that we only send user-specific notifications down that user’s connections. That’s straightforward as our data keys by device which maps to user. Also, communication is encrypted (wss is over TLS). We might also encrypt payloads end-to-end if sensitive, but since we control both ends (client app and server), TLS is sufficient.

State Management

Delivery workers are mostly stateless with respect to persistent data (they don't store data that can't be reconstructed), but they hold a lot of in-memory state (the connections). If a worker process crashes, all those device connections drop. The devices should detect that (TCP disconnect) and attempt to reconnect (likely to a different worker). This will temporarily delay messages to those devices. The system can handle it (some messages might fail to send because the connection was lost mid-send; they’ll be retried when the user reconnects). We should have health checks, if a worker is failing, remove it and have devices reconnect elsewhere. Possibly run multiple workers on each machine so that if one crashes, others still serve some connections.

Throughput and Parallelism

Each delivery worker can handle many messages serially if using async I/O because while one message is being sent, others can be processed on other connections. If a single device gets a storm of notifications, we may queue them locally or collapse them. But typically, different devices’ messages are independent. We ensure our consumption from Kafka is at least as many threads or async tasks as needed to utilize CPU cores. The NIC bandwidth might handle, say, 1 Gbps; if each message is 1KB, that’s ~1 million msgs per second possible per server theoretically. We probably won’t hit that; likely the bottleneck is elsewhere or we just won't need that many per server. But it indicates we have some headroom.

Logging and Metrics

After sending, the worker should log success or failure. As mentioned, updating the DeliveryLog table for each message might be heavy. One strategy: accumulate results and write in batches. Another is to send a log event to a separate logging service (maybe via Kafka) to be processed in bulk. For now, we assume the worker can do a quick update on the DeliveryLog row (which is indexed by primary key), a single row update is not too expensive, but at scale we’d prefer batch or asynchronous writing.

Platform Adaptations

If we consider an optimization: maybe for web we’ll use the standard Web Push protocol (with VAPID, etc.), which does require interfacing with browser push services (like Chrome’s push service). That would violate the “no external” rule slightly as we rely on browsers, but not as much as relying on FCM. If we did that, our Web worker would have to craft a push HTTP request to the browser endpoint. That’s an HTTP call (which could be done concurrently). It’s similar to how we’d call APNs for iOS in a normal scenario, except here it’s browser push endpoints. For completeness: we might incorporate that as needed, it doesn’t fundamentally change architecture, it just changes what Delivery Worker does for web (making an outgoing request rather than a direct socket send). The trade-off is we become dependent on those web push services which are third-party (Mozilla/Google push servers). To stick strictly to self-built, we continue with WebSocket assumption.

In summary, the Delivery Engine is designed to handle the fan-out to millions of devices reliably and fast. It uses persistent connections (sockets) to achieve real-time delivery (no additional latency of connecting per message). It must handle disconnections gracefully (triggering retries) and avoid message loss through acking and logging. By partitioning load and using asynchronous processing, it can scale horizontally, we can always add more worker servers to handle more devices or more messages. The architecture ensures each platform’s delivery is isolated (issues with mobile won’t stop web delivery, etc.) and that the engine can “manage its own rate limits, payload formats, retries, and auth” per platform. This modular approach means improvements or changes in delivery protocol (say we introduce a new protocol) can be done in that worker module without affecting the rest of the system.

Scheduler

The Scheduler component ensures that delayed notifications are sent out at the correct time, and it handles potentially large volumes of scheduled tasks in a reliable manner. Designing a scheduler in a distributed system requires careful consideration of timing accuracy, fault tolerance, and scalability.

Maintaining the Schedule

When a notification is scheduled (with a future timestamp), it needs to be recorded such that the system will remember to send it later. We do this by storing scheduled_at in the Notifications table. The scheduler service’s job is essentially to monitor this table (or a subset of data representing scheduled tasks) and trigger dispatch when due. We can implement this monitoring in a few ways:

Database Polling: Periodically (e.g. every minute or every few seconds), run a query like SELECT * FROM notifications WHERE scheduled_at <= now() AND NOT dispatched (with proper conditions) to find due notifications. For efficiency, we’d have an index on scheduled_at and perhaps a flag for dispatched. This is simple and leverages the DB’s consistency. However, querying large tables frequently can load the DB. We might optimize by only looking at the nearest upcoming entries (e.g., include AND scheduled_at <= now()+ interval '1 minute' to catch those about to fire, and keep track of ones just after). We could also poll less frequently if one-minute granularity is acceptable (some delay).
In-Memory Schedule (Timer Wheel): Load upcoming notifications into memory with timers. For instance, use a min-heap (priority queue) keyed by scheduled_at. The scheduler service can pop from the heap when the time is reached. But with potentially millions of items, keeping them all in memory might be heavy, and if the scheduler crashes, those are lost (so we’d still need to re-load from DB on restart). A compromise is to load only a window of tasks (say, tasks in the next hour) into memory, and as time moves, load more. This reduces DB hits at the exact moment but still needs a refresh.
Redis Sorted Set: As mentioned, we could push each scheduled notification ID into a Redis sorted set with score = scheduled timestamp. Redis can then efficiently give us “all items with score <= now”. The scheduler can do a ZRANGE/ZPOPMIN operation to retrieve due items. This is fast and can handle large sets if Redis has enough memory. It moves some load off the DB (though we have to insert into Redis when scheduling). If the scheduler dies, as long as Redis is running, the data is persistent enough (though in worst case if Redis lost data, the DB is still the source of truth).
Kafka delay processing: Another approach is to use a delayed message queue. For instance, produce the scheduled notification event to a Kafka topic with a timestamp such that consumers ignore it until that time (with Kafka, consumers can look at message timestamps). But Kafka itself doesn’t delay delivery; you’d need the consumer to filter. There are some libraries or using a separate scheduling service (like an advanced feature of some queue systems), but those get complex.

Given simplicity and reliability, we likely go with DB polling or Redis. For design, assume the scheduler polls the DB every few seconds for due notifications. We will ensure that query is efficient by filtering on scheduled_at and perhaps limiting how many it picks at once.

Ensuring Single Delivery

If we have multiple scheduler instances (for HA), we must prevent duplicate scheduling. Two instances might both find the same due notifications and enqueue them twice. To avoid this, we can use a simple DB row lock or update. One way: when scheduler picks up a notification to send, update a field “dispatched = true” or set scheduled_at = NULL (meaning it’s handled) or move it to a different table. If we do this in a transaction, the other instance either won’t find it (if we exclude dispatched ones) or if it does, it might attempt the same but one will update first and the second will see dispatched flag is now true. Alternatively, we elect a leader scheduler, only one actively queries. We could do leader election via a distributed lock (like using Zookeeper/etcd or even simpler, try to insert a row with a specific key in DB to act as lock owner). If leader goes down, others compete to become leader. This ensures only one scheduling process at a time. Given the complexity of distributed locks, an easier but slightly less robust approach is to rely on DB unique updates: e.g., mark each notification with a unique “scheduler_id” when taken. But let’s assume we will run one active scheduler instance and keep another on standby (failover scenario), as it’s simpler to reason about.

Accuracy

We aim to send at the scheduled time. If polling every minute, worst-case a notification might be up to 1 minute late. We can poll more frequently (every 10 seconds) for better accuracy at the cost of more DB load. Using a timer approach could trigger exactly on time. For typical notification use (like reminders, marketing), a minute or few seconds off is tolerable. For extremely time-sensitive stuff (like “execute a trade at this exact second”), you’d need a more precise scheduler with sub-second accuracy, but that’s beyond our scope.

Scalability

If there are a huge number of scheduled notifications, say 1 million scheduled for every 8:00 AM (like a daily job for each user), the scheduler will have to enqueue 1 million events at that time. It must be efficient at pulling those from DB and pushing to Kafka. Batch processing is key: we can retrieve in chunks (like limit 1000 per query, loop) to avoid locking the DB too long. Also, we might spawn threads to publish them. We should also be mindful of memory, don’t load all million into memory at once if possible, stream them.

Scheduler and Campaigns

For a campaign to many users scheduled at once, the scheduler might alternatively drop a single “campaign start” message and rely on routing service to do the heavy lifting (distributing to all users). That could be more efficient than the scheduler itself doing all fan-out because the router might be more distributed. We could design it such that for a broadcast, the Notification record indicates a broadcast, and scheduler just sends a message "broadcast notification 999 now". All router instances see that and each takes a portion of users. This requires coordination in routing (like each router instance takes every Nth user or so). It’s an advanced optimization to distribute load. If not implemented, the scheduler can also directly perform a big DB query: “SELECT all device tokens” and loop to enqueue, which might be heavy but one central place. Depending on where we expect the bottleneck, we can adjust.

Fault Tolerance

If the scheduler crashes mid-process (say it pulled 1000 notifications to send, enqueued 500, then died), those 500 might not be marked dispatched. On restart, it might resend them. This could cause duplicates. We mitigate by marking before enqueue or using transactions: perhaps mark them dispatched, then if crash, some marked as dispatched but not sent, those would be lost unless we have a recovery to find dispatched but not actually in logs. To handle this, we might actually rely on at-least-once and deduplicate at the consumer side via the unique notification_id + user pair. But deduping that at scale is tricky. Instead, a simpler approach: it’s okay if a user gets a duplicate notification in rare crash scenario (not ideal but not the end of world for most use cases). Or we design an audit job that cross-checks: e.g., after sending, verify how many were delivered vs expected, and if some missed, try again.

We strive for exactly-once scheduling by careful marking. One possible implementation: use the database’s own ability to schedule. Some DBs allow scheduling a job or we could use a cron-like external scheduler (like Quartz with a DB job store or AWS EventBridge). But building it ourselves is educational and flexible.

Given these considerations, our Scheduler service will likely:

Use the database as the primary store of schedules.
Poll at a fixed interval for due notifications.
Lock or mark them when processing to avoid double-sends.
Push to the message queue for actual delivery.
Handle failure by either relying on the database state or having an administrative way to re-run a window if needed.

This component ensures we meet the requirement of delayed notifications and does so in a scalable way. It is effectively a specialized producer to the notification pipeline, triggered by time instead of an immediate API call. By decoupling scheduling logic from delivery logic, we can scale and tune it without affecting the real-time flow.

Retry Mechanism

Despite our best efforts, some notifications will fail to be delivered on the first attempt. Common reasons include devices being offline (no active connection), transient network errors, or internal errors. The system should automatically retry those notifications to maximize the chance of delivery, while avoiding infinite retries or overwhelming the system.

When to Retry

We decide to retry on any transient failure. This includes:

Device connection not available or timed out.
Network issues sending to device (e.g., broken pipe).
Possibly if using external push (not in our main design, but say web push HTTP returns a 500 or timeout).
A worker crash or outage that prevented completion (the message would requeue automatically if not acknowledged). We do not retry on permanent failures:
If a device token is invalid (in our system, perhaps if the app uninstalled and never reconnects, we might eventually give up).
If user has uninstalled app (we would learn that only if device doesn’t reconnect for long, or if we had a way to detect).
If our payload was bad (though that should be caught earlier).

How Retries are Enqueued

The moment a delivery attempt fails, the responsible Delivery Worker (or a monitoring process) should schedule a retry. One straightforward implementation is:

On failure, update the DeliveryLog attempt_count and status = 'pending_retry'.
Then produce a new message onto a Retry Queue with information about the notification and device. However, we don't want to retry immediately, we want a delay.
If using RabbitMQ, we could set a TTL on the message (e.g., don’t make it visible for X seconds).
If using Kafka, we might produce to a separate topic like "retry_1min" which consumers ignore until 1 minute later (maybe by storing the target execution time in the message).
Another approach: have the worker set a timer or push to the Scheduler (like "deliver notification 123 to device X at now+1min").

A simpler design: use the same scheduler/Redis mechanism for retries. For example, have a Redis sorted set for retries. On failure, add entry (device X, notif 123) with score = now + backoff_time. A Retry Handler service (which could be part of the scheduler or separate) will poll that sorted set for due retry entries and then move them to the normal delivery queue. This is conceptually clean: treat retries as scheduled tasks.

Backoff Strategy

We use exponential backoff to avoid hammering a down device. For instance:

1st retry after 30 seconds,
2nd after 5 minutes,
3rd after 30 minutes,
4th after 2 hours, etc. We cap the maximum delay, say maybe 4 or 5 attempts over a few hours. If after, say, 4 attempts (total maybe ~2.5 hours), still not delivered, we mark it as permanently failed (status 'failed' in log) and stop retrying. By then, if the user hasn’t come online, that notification might be moot (they might see content when they open the app anyway).

Avoiding Duplicate Sends

With retries, there is a risk that a device comes online and gets the second attempt while the first attempt was actually delivered late or something. To handle this, we ensure each notification to a device has a unique ID and we track if it was delivered. The DeliveryLog serves this purpose. Before sending, a worker could check “is status already delivered?. If yes, skip. But normally, if delivered, we wouldn’t be retrying. Another scenario: after scheduling a retry, the user comes online and the device fetches pending messages manually (if that was a feature), then our scheduled retry might still attempt to send it again. Deduplication either at server or client side can help. We can include a unique notification ID in the payload so the client can ignore duplicates if it receives any.

Dead Letter and Alerts

If a notification ultimately fails (all retries exhausted), we mark it as 'failed'. We might move its record to a Dead Letter Queue (just for analysis). Maybe a monitoring system tracks these failures. For critical notifications (security alerts, OTPs), maybe we have a fallback mechanism, e.g., send an SMS or email if push failed. That’s an application-level decision and could be a future enhancement. At least, we log it so that support or analytics knows that user didn’t get that notification.

Rate Limiting Retries

We have to be careful that a mass outage (e.g., our connection servers go down) doesn’t cause a storm of retries that overwhelms the system when it recovers. If 5 million devices all disconnect, we should not instantly retry all 5 million notifications 30 seconds later. That could flood the system. Ideally, the retry schedule is per notification/device, so they’re spread out. But a big event can still cause many to coincide. We might implement jitter (random small offsets to spread retries) and global rate limits: e.g., only process X retries per second. This might slightly delay some notifications but prevents overload.

Example Retry Flow

Bob’s iPhone was offline in our earlier scenario, the attempt at 10:00 failed. The worker sets attempt_count=1 and schedules a retry after 5 minutes. At 10:05, the retry mechanism enqueues the message again. Meanwhile Bob turns on his phone at 10:06 and connects. The retry message is picked by a worker at 10:06, it finds device connected now, sends the notification, and succeeds. Bob finally gets it. If Bob hadn’t come online, perhaps another retry would happen at 10:30, then maybe 12:30, then finally give up. Each attempt is recorded.

Implementation details

We might incorporate the retry logic into the Delivery Worker itself up to a point, e.g., the worker could do a couple quick retries locally (like try again after 5 seconds if a minor issue) but for longer delays, better to offload to a scheduler. Using the same infrastructure (like our scheduler or a delayed queue) ensures consistency. RabbitMQ’s dead-letter exchange is a neat solution: failed message goes to a dead-letter, which is configured to route to the original queue after a delay. This can implement backoff by increasing the TTL each time (perhaps encoding attempt in message). With Kafka, since it doesn’t natively support delays, a common approach is multiple topics: retry1, retry2, etc., with consumers set up to only read after certain time, or application-managed delays.

Given our design, a unified approach using a sorted set or the existing scheduler (with some code changes to handle device-level entries) might be simplest. We could also combine it with the main DeliveryLog table: the scheduler could look at DeliveryLogs with status 'pending' and scheduled retry timestamps, but that complicates the DB usage. Better keep retries logic outside main DB to reduce load.

User Feedback

The system doesn’t directly inform the end-user of retries, it’s internal. But perhaps internal monitoring might alert if many retries are happening (could indicate an outage or bug).

In essence, the Retry mechanism ensures “no notification is lost” easily, if a first attempt fails, we try again later, embodying the reliability requirement that we prefer delayed delivery over none. By implementing exponential backoff and a limit, we also avoid infinite loops or spamming. This mechanism, combined with the DeliveryLog tracking, provides at-least-once delivery semantics with eventual success in most cases, and we guard against duplicate delivery via unique IDs and status check.

Trade offs/Tech choices

In building this push notification service, we made several technology and design choices, each with pros and cons. Let’s discuss some key trade-offs and the rationale behind them:

Custom Push vs. Platform Push

We chose to implement a custom push channel (WebSockets) instead of using APNs/FCM, as required. The trade-off here is control vs. complexity. By going custom, we have full control over the push pipeline (no external dependency, flexible data format, unified logic for mobile/web) but we sacrifice the convenience and battery-efficiency of the native push services. For instance, on iOS, not using APNs means our app must maintain a background connection which can be power-hungry and may not be allowed to persist long if the app is not in foreground. This could lead to missing notifications if the app is killed. On Android, a custom service is more feasible (some messaging apps use their own push). We accept these limitations in exchange for a self-contained system. Technology-wise, we picked WebSocket (over TLS) for its ubiquity and ability to send push data in real-time. Alternatives considered: MQTT (a lightweight pub-sub protocol) which is very efficient for mobile push and has features like persistent sessions. MQTT could reduce overhead, but it requires running an MQTT broker or implementing one, that’s an extra component. WebSockets can be handled by our existing web servers and give us a straightforward message channel. So we went with WebSockets for simplicity of using HTTP/TLS infrastructure.

Message Broker – Kafka vs RabbitMQ

For the internal queue, we favored Apache Kafka due to its high throughput and durability characteristics, especially given potential burst traffic. Kafka’s partitioning allows us to scale consumers easily and handle ordering per key. It’s also very durable (writes to disk, replicate) which aligns with our “never lose a notification” goal. The trade-off is that Kafka is a bit heavier to operate and doesn’t natively support delayed messages or straightforward topic-based routing (it’s more stream-oriented). RabbitMQ offers easier routing (topics, direct exchanges) and features like TTL for delayed retries, and it’s simpler for real-time use cases up to a point. However, RabbitMQ might struggle with extremely high rates (tens of thousands of messages per second sustained) without complex clustering, and storing millions of messages can exhaust memory unless carefully configured. Kafka shines in letting us backlog millions of messages on disk with sequential I/O. So we chose Kafka to future-proof for very high scale, with the understanding we’d implement our own retry delays or scheduling on top. We also considered that Kafka integrates well with big data pipelines (for logging, etc.), which might be a plus if we later analyze notification events.

Database – SQL vs NoSQL

We decided on a SQL relational database for core data (devices, notifications, logs) due to its strong consistency and relational queries. This makes it easier to enforce uniqueness (one device record per token) and perform joins (like linking notifications and deliveries for stats). The drawback is scaling writes and storage for the DeliveryLogs table. A NoSQL solution like Cassandra could handle a massive scale of writes and multi-datacenter replication more easily, at the cost of complex query logic (Cassandra is good for time-series inserts and lookups by key, but less so for flexible queries). We stuck with SQL for now, leveraging partitioning and possibly sharding to scale. If we hit limits (e.g., cannot write 100k rows/sec), we might then consider moving the log storage to a NoSQL or splitting the load. The choice also hinges on team familiarity and simplicity, SQL is straightforward for ensuring correctness early on. Additionally, by using SQL, we can use features like foreign keys and transactions to ensure, for example, when we insert a Notification and multiple DeliveryLogs, it’s atomic (all or nothing), which helps maintain consistency.

Cache – Redis

We incorporate Redis as a caching layer for quick device lookups and possibly scheduling. This is a conscious choice to relieve load from the SQL database. The trade-off is adding another component to maintain and potential cache inconsistency if not properly invalidated. But since device data changes infrequently, cache consistency is manageable (update cache on device add/remove). Redis gives O(1) or O(log N) set operations which is hugely beneficial for quick fan-out selection (like getting all devices of a user or union of segments. We accepted the added complexity of a cache for the performance benefits.

Consistency vs Availability

In certain places we prioritized durability (consistency) over immediate availability. For example, we write to the DB then queue, even though that adds a write and could slow down the API slightly, because we want that persistence in case of crash. We also ensure the broker is durable. One might argue for an even more available system: e.g., just fire the notification to memory and hope a worker picks it up, for lower latency. But then a crash loses data. We chose the safer route: every notification is persisted (at least in the queue if not DB, but in our design both). The trade-off is a bit more latency and complexity (managing DB and queue transactions). We deem reliability more important for this system (especially if notifications include critical info like security alerts).

Processing Model – Synchronous vs Asynchronous

We heavily use asynchronous processing (via queues, background workers). The benefit is decoupling and throughput, but it complicates knowing the end-to-end status. For example, when an API call returns, the message might not be delivered yet. If the client wants to ensure it was delivered, they have to poll or get a callback. We accepted this complexity because synchronous processing of a notification to millions of devices is impossible, it must be async. Even for one device, doing it sync would tie up threads waiting on network. So async was the only viable option at scale. We just need to provide hooks (like the GET status API) for tracking if needed.

Partitioning Strategy

For Kafka, we had to decide on partition keys. We likely partition by user or device ID so that order is preserved per user/device. Why is ordering important? Possibly if two events occur back-to-back for the same user, they should receive them in sequence. If they were processed out of order, it could confuse (imagine “Order shipped” and “Order placed” arriving inverted). Partitioning by user ensures that won’t happen. The trade-off is one user’s messages all go to the same partition/worker, which is fine; with millions of users, keys distribute fairly. If instead we partitioned randomly, we’d maximize parallelism, but lose ordering, which could be problematic. We chose to maintain ordering per user as a correctness measure. Also, partition by user means we can also accumulate user-level tasks or data in one place if needed (like easier to gather “all pending notifications for user” if needed by looking at one queue partition, though that’s a minor point).

Language/Framework

While not explicitly required, the implementation language is a choice. High-performance components (like the Delivery Worker) might be written in something like Go or Java for speed and concurrency. The trade-off might be development speed vs performance. If the team is well-versed in Java, using its ecosystem (Netty, Spring for API, etc.) might be fastest to build reliably. If low-level tuning is needed, perhaps Rust or C++ for the push server could reduce latency, but at cost of complexity. We probably will go with a popular server language (Java, Go, or Node) for easier hiring and maintenance. It’s a trade-off between absolute performance and maintainability. Considering scale, Java or Go can more than handle 10M connections with proper architecture, so we don’t need to dive into exotic tech.

Logging and Analytics

We chose to log events in a relational DB (DeliveryLog). An alternative is to use a logging pipeline (like sending all events to a Kafka topic “notifications_log” and then consuming that into a data lake or real-time monitoring system). That alternative might be more scalable for analytics, but less straightforward for quick queries (like our GET status API can easily query the DB, whereas if logs were only in Kafka, we’d need a separate store or to search the stream). So we opted to keep logs in SQL for now for simplicity of queries, acknowledging the scale issue. A compromise could be to log to both: DB for short-term status, and long-term data to a big data store. That might be a future improvement once volume grows.

Our choices aim to balance scalability, reliability, and complexity. We leaned towards proven scalable components (Kafka, Redis, stateless services) even if they add setup overhead, because the user base size demands it. We maintained strong consistency in the core data to avoid edge-case bugs (like missing or duplicate notifications). Each trade-off was decided by asking: does it help us handle 10M users and spikes of traffic without crashing or losing data? If yes, we likely chose that even if it’s harder to implement. This results in a system that “embraces a distributed, decoupled architecture that can fan out, throttle, retry, and audit billions of messages with minimal bottlenecks, aligning with the best practices for high-scale notification platforms.

Failure scenarios/bottlenecks

No system is perfect; we must anticipate possible failure modes and performance bottlenecks, and design mitigations for them. Here we discuss several such scenarios and how our design handles them:

Database Failure or Slowdown

The database is central for storing devices and logging notifications. If the primary database goes down unexpectedly (hardware failure, crash), our system might not be able to accept new notification requests (since we can’t persist them or fetch devices).

To mitigate this, we employ a replicated database setup, e.g., a hot standby or a cluster (for PostgreSQL, perhaps using streaming replication to a standby). The system can promote a standby to primary within seconds via automation. During the failover, the API might briefly reject requests (or queue them in memory) until the DB is back. Non-functional requirement of 99.99% uptime means we aim for minimal downtime: a failover can often be < 1 minute. Additionally, for reads (like device lookup), we use replicas so that even if the primary is overloaded or momentarily down, the workers might still fetch device data from a read replica (which is still up to date within seconds).

If the entire DB cluster is down (network partition, etc.), that’s a critical failure, the system will be mostly stalled (though in-flight queued notifications could still be delivered, since workers have them). As a safety, our message queue is decoupled, so it can still deliver already queued messages even with DB down. For example, if DB is down but Kafka is up, notifications that were already in Kafka get processed by workers using cached device info. However, if workers need to query DB for device info and can’t, those specific sends might fail (they could retry later when DB is up). We alleviate this by heavy caching: workers should have most device tokens they need either preloaded or cached from previous queries, reducing dependency on the DB per message.

Message Broker Failure

If our Kafka cluster has an outage (e.g., a majority of brokers crash or the cluster becomes inquorate), new notifications cannot flow through the pipeline, and that is a critical failure. We mitigate by using Kafka’s replication and running it on reliable servers (maybe even across data centers if possible). A partial outage (one broker down) is handled by Kafka automatically by electing new leaders for partitions; consumers/producers might see a brief pause but then continue. We should configure a sufficient replication factor (e.g., 3) so that one broker down doesn’t lose messages.

If Kafka is completely down, our API might start buffering notifications in memory or a fallback store, but that’s risky, likely we’d respond with an error (“service unavailable”) to the API calls so callers know to retry later. So, high availability of Kafka is vital. We also consider using dead-letter queues for Kafka: if a message can’t be processed after many retries, it might go to a separate topic to not clog the main one. The reference architecture emphasizes that each component should have isolation; Kafka helps by acting as a buffer, but if it fails, it’s indeed a single point that stops the pipeline. Running Kafka in a multi-zone setup and monitoring it closely is our strategy. In extreme case, if Kafka is down long, we might manually trigger our backup plan: e.g., have the API log unsent notifications to a file or DB table to replay later.

One of the Microservices Crashes

Router Service Crash: If the routing service crashes or is deployed buggy, notifications might pile up in the initial queue (notifications_to_send) and not get fanned out. But the messages are safe in the queue. We likely run multiple router instances; if one crashes, another can take over processing those partitions (Kafka consumer group rebalance). So a single crash just reduces capacity temporarily. If all router instances crash due to a bad deployment, the queue backlog grows. We’d notice from monitoring (no messages getting through). The fix is to redeploy a good version quickly. The system can recover by processing the backlog when routers are back, albeit with delay. This is acceptable as long as downtime isn’t too long.

Delivery Worker Crash: If a delivery worker (or an entire group, like all iOS workers) crashes, then devices of that type won’t receive messages for that period. However, thanks to the message broker, any unprocessed device messages remain in the queue. We could even have other workers pick them up if we configure cross-reading (or spin up new workers). The design isolates platform queue, which is good to not affect others, but it also means if all iOS workers die, iOS messages sit until iOS workers recover. Our mitigation is to have health checks that restart crashed workers quickly (maybe orchestrated by Kubernetes which restarts containers automatically on failure). Also, running N>1 workers for each platform ensures one crash doesn’t zero out capacity.

Scheduler Crash: If the scheduler service goes down, scheduled notifications won’t be dispatched at their times. If it’s down for a short time, it will catch up on missed notifications when it restarts (it can scan a window in the past). If down for a long time, many schedules are missed, this is bad. We avoid that by having a standby or making the scheduler a highly reliable process (maybe less frequently updated, simpler logic). Leader election or even running two in parallel (with a locking mechanism on each schedule item) can give redundancy. The reference suggests blue-green or canary deployments for stateful services like scheduler to avoid downtime during deploy.

Device Gateway Crash: If a server holding many device connections crashes, all those devices disconnect. This is somewhat expected in large systems, devices will attempt to reconnect (with exponential backoff perhaps to avoid thundering herd). The immediate effect is any in-transit notifications to those devices fail. Our retry mechanism will catch those. We mitigate impact by distributing devices across many servers so that one server’s failure only affects, say, 1% of devices, and by having clients implement reconnect logic. Additionally, we might use a load balancer that quickly stops sending new connections to the dead server and shifts them.

Network Partition / Latency

If network issues occur between components (say, API can’t reach database, or workers can’t reach Kafka), parts of the system may stall.

For example, if workers can’t reach DB, as mentioned, they might fail device lookups. We have local caches so they can still deliver known devices, but any new user device mapping might not be found. If API can’t reach DB, it can’t record new requests, it might then either reject them or operate in a degraded mode (maybe queue them in memory with risk). It’s safer to return error to callers than silently drop. For network partitions, one strategy is to design for graceful degradation: e.g., if DB is unreachable, maybe only allow sending to cached devices, or if cache is still warm, continue processing what we have. However, since consistency is important (we want to log everything), we likely err on halting certain operations until network is restored. We do ensure to separate networks: internal traffic (between microservices) might be on a fast LAN, and external traffic separate, to avoid external spikes affecting internal comms.

High Load / Bottleneck

Suppose an extremely large campaign triggers 20 million notifications at once. Potential bottlenecks:

Broker throughput: Can Kafka handle, say, 20 million messages in a short time? If we have enough brokers and partitions, yes, but if misconfigured, it could throttle or cause high disk IO. We mitigate by provisioning Kafka with sufficient throughput (e.g., using SSDs, multiple nodes) and possibly compressing messages to reduce bandwidth. Also we can tune producer batching to make better use of IO.

Router and Worker CPU: Fan-out of 20 million might strain CPU if not optimized (string manipulations, JSON encoding etc.). We should profile and possibly use efficient languages. We can also scale out horizontally: more router instances to share the load. Since each runs in parallel on different partitions, adding more should linearally increase capacity until other limits (like DB or network) hit.

Database writes: 20 million notifications to all users means 20 million delivery log inserts. If done in a short time, that’s a huge write spike. A single DB server likely cannot sustain that many inserts per second. This is a big potential bottleneck. Mitigations: use batching (e.g., insert multiple rows in one SQL command), use partitioning to avoid index contention, or offload bulk inserts to a background job to avoid slowing the sending loop. We could also consider dropping some logging for such mass events if it’s not critical to log each device success (maybe log aggregated result). However, for consistency we planned to log each. We might therefore resort to scaling the DB vertically (a very powerful machine) or horizontally (shard logs by user range across multiple DB servers). This is a complex area; a possible improvement is to separate the concern: have a specialized logging storage (like Kafka or Cassandra) for high-volume logs instead of a single SQL table.

Memory usage: If routers or workers try to load too much data (like reading all device IDs for 10 million users in memory), they might run OOM. We must implement streaming (process in batches rather than load-all). Use of generators or pagination when dealing with large loops is necessary. Also, our caches (like Redis) should be sized properly to hold all active devices if we intend to cache them.

Notification Storm / Abuse

If a buggy client or malicious actor tries to send a huge number of notifications (spam), it could overwhelm parts of the system. For example, if someone found a way to call the API to send 1 million notifications repeatedly. We mitigate with rate limiting at the API (e.g., an API key can only create 100 notifications/minute or whatever fits normal use). Also, internal systems need guardrails (maybe the product only triggers notifications under certain conditions). We can also implement per-user rate limits in the router to avoid flooding a user with too many messages in a short time (to protect user experience. This is more of a logical throttle than a system stability one, but it also protects the system (if we accidentally tried to send 1000 notifications to the same device in a second, that could be wasteful or cause performance issues on that device’s connection).

Cascading Failures

An example of a cascading issue would be: database gets slow -> API requests pile up -> thread pools fill -> API response timeouts cause callers to retry -> even more load -> system meltdown. To avoid this, we must implement backpressure and timeouts. The API should quickly timeout if DB isn’t responding, and return error, not hang. Also use circuit breakers: if we detect DB is down, perhaps fail fast on API calls rather than tying up resources. Similarly, if workers see Kafka lag growing, they can slow down intake or drop non-critical messages. Partition isolation helps; e.g., if iOS push is failing and backlog builds, it shouldn’t block Android push (we separated those queues). Each part can then fail somewhat independently rather than whole system.

Data inconsistency

Perhaps not a crash, but if code bugs cause inconsistent data (like duplicate device entries or a notification not logged properly), that could confuse the system. We rely on database constraints (unique indexes to avoid duplicate device, foreign keys for referential integrity) and also implement idempotency (the event IDs to avoid duplicate sending. In worst case, duplicates mean a user might get the same notification twice. That’s undesirable but not as bad as not getting it at all or system crash. We have monitoring for duplicate sends (maybe compare event IDs in logs). If it happens, we fix the bug.

In designing for failures, we strive for graceful degradation. For example, if the scheduler fails, immediate notifications still work; if one platform’s push fails, others still function. In the worst case that the entire push system is down, the rest of the application remains up (just without notifications), this is important too (don’t let push issues take down the main app). We isolate resources: e.g., the push system might use separate database instances and caches than the core app, so if push is overwhelmed, it doesn’t starve the core app’s database.

We also employ robust monitoring and alerting: track queue lengths, processing rates, error rates, DB load, etc. If any metric goes beyond threshold, alerts go out and automated scaling might trigger (e.g., auto-add Kafka consumers or spin up more worker pods. For instance, if the “pending notifications” queue grows and isn’t reducing, that indicates a downstream bottleneck. We could automatically scale out workers in response or at least alert engineers to investigate (maybe some workers hung or crashed).

To test these scenarios, we would run chaos tests: e.g., kill a database node, drop some connections, flood with traffic, etc., to see that the system recovers without data loss. Tools like fault injection (as referenced in chaos testing) can simulate these events to verify our mitigation strategies.

Overall, by design:

There is no single point without a failover or redundancy (DB replicated, Kafka clustered, stateless components duplicated).
The system can queue up backlog if parts become slow, rather than dropping data (giving time to recover or scale.
Each component can be scaled if it becomes a bottleneck, and we’ve identified likely ones (DB and logging, fan-out throughput) for special attention.
We’ll implement timeouts, retries, and fallbacks at integration points to avoid one slow component hogging resources.

This preparation ensures that even under heavy load or partial outages, the system remains fault-tolerant and resilient, meeting the high availability and reliability requirements expected of a notification service.

Future improvements

Having built a robust push notification service for 10 million users, we can consider several improvements and extensions to enhance the system further as needs evolve:

Scalability Enhancements

As the user base grows (e.g., 50 or 100 million users) or traffic patterns change, we will need to scale out infrastructure. We can invest in multi-region deployment, deploying push service clusters in data centers around the world. This would shorten the network path to users (reducing latency) and provide redundancy (if one region fails, others can take over). It introduces complexity in routing (we’d need to route each user’s device to a home region).

We could use DNS-based routing or anycast IP for WebSockets so that devices connect to the nearest server. The message broker could be partitioned by region or use a cross-region replication (Kafka has MirrorMaker to replicate topics across clusters). Another area is the database sharding: splitting the user/device data across multiple database instances (for example, shard by user ID range or by geography) to handle more users. We’d need a mechanism to route queries to the correct shard. This would allow essentially linear scaling of the user base. Likewise, the DeliveryLog table might be sharded by date or user to keep each shard manageable.

As we scale, automation in orchestration (like using Kubernetes for auto-scaling pods, and maybe auto-scaling the number of Kafka partitions) will become important to handle spikes (e.g., a celebrity sends a message that causes a fan-out to millions in seconds). We might implement elastic scaling: monitor queue delays and automatically add worker instances or allocate more CPU to the router, then scale back down when load subsides.

Personalization and Intelligence

Currently, our system treats notifications somewhat uniformly. Future iterations could integrate with user data to personalize content and sending times.

For example, Personalized Send Time, instead of blasting all users at 9 AM, we could stagger notifications based on when each user is most active. This could increase engagement and also avoid traffic spikes. We might mine user behavior to see that user A typically opens the app in the evening, so send their daily notification at 6 PM, whereas user B checks in the morning, so send at 9 AM. Implementing this would involve analytical components and possibly moving scheduling logic closer to the user: e.g., store a preferred send hour for each user and have the scheduler respect that. Another aspect is content personalization: If it’s a notification like “Recommended products,” we could have the notification service fetch personalized content for each user.

That is more of an application-layer concern (likely an external service provides the content for each user ID), but our pipeline could support it by allowing a callback or template rendering step. We’d add a Template Service or integrate the router with a template engine that fills in user-specific fields (like name, or item count) before sending. We might also incorporate A/B testing in notifications, sending slightly different messages to different random sets to see which performs better (click-through, etc.). That requires our system to be able to randomize and track experiment groups, which could tie into the preferences or a new experiment service.

Multi-Language Support

As the user base becomes global, supporting multiple languages in notifications is key. We should enhance our notification creation process to handle localized content. This could mean when a notification is created, it includes translations for various languages (or an identifier to fetch the translation). The system would need to know each user’s preferred language (likely stored in user preferences). Then, before delivery, the content should be in the correct language for that user. We could accomplish this by storing a mapping in the Notification record like message_en, message_fr, etc., or storing a template ID and having a translation service. The router or delivery worker can then choose the appropriate field based on user’s language. This adds memory overhead (multiple strings) but is straightforward. Alternatively, we generate distinct notifications per language group behind the scenes and target those users. Regardless, the improvement ensures users get messages they can read, greatly improving effectiveness. It’s a trade-off of more data handling vs user experience. Given modern requirements, we’d plan for at least a dozen languages if 10M users are worldwide.

Additional Channels

Our design could be extended beyond push notifications to other channels like email or SMS using the same architecture. We already handle scheduling, fan-out, etc. We would introduce new worker types, e.g., Email Delivery Workers that take notification events and send emails via an SMTP server or email API, and SMS workers using an SMS gateway. The message format might need additional fields (email subject, phone number, etc.), but structurally it fits. We might need separate queues for email and SMS to isolate them as we did for push types. By doing this, our service becomes a unified notification platform (multi-channel messaging), which many systems aim for. The functional requirement would expand to ensure one event can trigger notifications on multiple channels according to user preference (like if user prefers email for certain alerts, etc.). We’d then definitely need more preference settings (which channel to use for which user or type). The benefit is a single system can orchestrate all user messaging.

Enhanced User Preferences and Control

In the future, giving users more fine-grained control is valuable. For instance, allowing users to specify quiet hours (already planned), frequency caps (“don’t send me more than 3 notifications per day”), or topic subscriptions (choose which categories of notifications they get). Implementing this would deepen our Preferences subsystem. The router (or even the ingestion phase) would have to respect these rules, e.g., if a user already got 3 notifs today, the 4th one is dropped or queued for next day. Or if user unsubscribed from “news” category, and a news notification comes, we skip them. This requires maintaining counts (could be done by incrementing a counter in cache per user per day and checking it). Also possibly scheduling deferrals (like “deliver after quiet hours end”). We have basic support for quiet hours in preference, but we might integrate that with the scheduler to automatically adjust scheduled_at for those users.

Improved Monitoring and Analytics

We can build a richer analytics dashboard: tracking delivery rates, open rates (if we instrument the app to report opens), conversion (if the notification was about an offer, did user click it?). This likely means funneling our logs to an analytics system (like ElasticSearch or BigQuery) and performing analysis. We might implement real-time monitoring of important metrics, e.g., a live graph of notifications sent per minute, success rate per platform, etc. This helps in operations and also in proving the system’s value (e.g., to product owners, showing how notifications drive user engagement). We might integrate a feedback loop: for example, on iOS, APNs has feedback for invalid tokens; in our case, since we don’t use APNs, we might still want to detect “device uninstalled app” if possible (maybe if device hasn’t connected in 3 months, assume uninstalled, and stop sending or purge that device). This can be part of a cleanup job that improves data quality over time.

Optimization and Tech Refresh

Over time we may find bottlenecks where new technologies could help. For instance, if managing millions of WebSockets becomes too memory heavy in our chosen language, we might adopt a specialized push server (some companies build custom servers in C++ or use event-driven servers like Nginx’s push module). Or if our database writes are struggling, we might integrate an event sourcing approach where notifications are an append-only log and then use stream processing to update read models. That could handle scale better but is a big change. We could also explore Kafka Streams or similar to handle some processing (like using stream joins to attach user preferences to events in real-time instead of DB lookups). Such changes would be driven by profiling and usage patterns observed in production.

User Inbox Feature

A future feature could be a persistent inbox where users can see past notifications in the app. This would require storing notifications per user in a queryable way (which our DeliveryLog partially does, but we'd probably make a separate collection for user-facing messages). We might build an API for the app to fetch last N notifications. This could improve experience (if a user dismisses a notification and later wants to find it). Implementation might involve moving delivered notification content into a fast key-value store keyed by user (like Redis or DynamoDB) for quick retrieval, or leveraging the existing logs if efficient. It’s an additional load, but manageable if done with care.

Auto-Scaling and Cost Optimization

As we scale, cost of running so many servers and processes becomes an issue. We’d implement auto-scaling to turn off some workers during low load (e.g., midnight if traffic is low) to save resources, and scale up in daytime or events. Also, optimizing code to handle more load per server reduces how many servers we need, saving cost. We might invest in more efficient serialization (binary protocols instead of JSON, etc.) to cut network and CPU overhead. Over time, these micro-optimizations add up.

Submissions

Displaying Solutions with Scores Ranging from 1 to 10