Interview opener

“I’ll anchor the design on invariants first, then walk through requirements, API semantics, data model and consistency, HLD, critical flows, scaling, failure handling, tradeoffs, and observability.”

That already signals senior/principal thinking.

Invariants (anchor the entire design)

These are the rules that should remain true even under retries, provider failures, backlog, and node crashes.

Core invariants

1. Protect user experience by priority
   • Critical notifications must stay low-latency and should not be delayed behind promotional traffic.
   • Promo can be throttled, delayed, or dropped if expired.
2. Never send expired promotional notifications
   • If the promo window is over, it must not be delivered.
   • TTL cleanup alone is not enough for correctness.
3. No duplicate user-visible sends
   • Retries, scheduler restarts, and queue redelivery must not cause duplicate notification delivery.
4. Bounded retries, bounded queues, bounded fanout
   • Avoid retry storms and fanout explosions.
   • No unbounded expansion of a campaign into millions of execution records at once.
5. Debuggable end-to-end
   • Every notification has stable IDs:
     • notification_id
     • campaign_id (if promo)
     • execution_id
     • request_id
     • provider_request_id
6. Downstream-aware throttling
   • Providers (email/SMS/push) have limits and failures.
   • Internal queues must not overwhelm providers or each other.

Why these matter

These sound generic, but the value is in the mechanisms they force:

• expired promo invariant → enforce at scheduler and worker
• no duplicates → idempotency keys + outbox + provider dedup if available
• bounded fanout → controlled segment expansion, not one giant blast

That is what makes this high-signal in interview.

1) Requirements (Functional + Non-functional + SLOs)

1.1 Functional requirements

Core features

1. Accept notification requests
   • Critical:
     • usually direct per-recipient (e.g., 1:1 message, security alert)
   • Promotional:
     • often system-generated and can target large segments
2. Support multiple channels
   • push
   • email
   • SMS
   • possibly in-app/websocket as an internal fast channel
3. Support priority classes
   • Critical
   • Promotional
4. Support expiration
   • especially for promotional notifications
   • if it misses its valid delivery window, drop it
5. Retry transient failures
   • provider timeout, 5xx, temporary 429, etc.
6. Avoid duplicate delivery
   • across retries, requeues, scheduler failover
7. Record delivery state
   • queued
   • sent
   • failed
   • dropped (expired)
   • suppressed (dedup/opt-out/rate limit)
8. Handle fanout
   • campaigns can target thousands to millions of users

1.2 Non-functional requirements

1. Scale
   • target 1M notifications/sec
   • with 80/20 critical/promotional mix
2. Low latency for critical
   • chat-like notifications must be near-real-time
3. Isolation
   • promo traffic must not degrade critical traffic
4. Reliability
   • tolerate worker crashes, queue retries, provider partial outages
5. Correctness
   • no duplicate send
   • expired promo not delivered
6. Extensibility
   • easy to add channels/providers
   • easy to add per-tenant or per-campaign quotas
7. Cost efficiency
   • avoid unnecessary writes/fanout bursts
   • avoid over-notifying when campaigns are stale or provider-limited

1.3 SLOs

You can state them like this:

Critical path SLOs

• low queue age
• low end-to-end latency (enqueue → provider handoff)
• very high delivery attempt success

Promo path SLOs

• eventual delivery within campaign window
• strict “do not send after expiration”
• partial completion acceptable if window closes

Operational SLOs

• bounded queue delay
• bounded retry amplification
• no cross-priority starvation

You don’t need exact numbers unless interviewer asks, but it helps to say:

“Critical and promo have different SLOs, so I separate both data flow and worker budgets.”

That’s a strong Principal framing.

2) API Surface (how producers interact with the system)

A notification system often has an ingestion API plus internal scheduler/worker interfaces.

2.1 External ingestion APIs

A) Create critical notification

Example:

POST /notifications/critical

Request:

• recipient(s)
• channel(s)
• payload/template vars
• dedup key (optional but recommended)
• available_until (optional for critical, e.g. typing indicator)
• tenant_id
• priority = critical

B) Create promotional campaign

Example:

POST /notifications/promotional

Request:

• campaign_id
• segment reference or recipient source
• channels
• template
• personalization source
• start_at
• available_until
• tenant_id
• rate caps / budget caps

C) Cancel campaign

Example:

POST /campaigns/{campaign_id}/cancel

This matters because promo queues may already contain work.

2.2 Internal APIs / contracts

Scheduler / fanout interface

The scheduler expands campaigns into controlled batches of recipient work.

Worker / provider interface

Workers call providers and classify results:

• success
• permanent failure
• transient retryable
• throttled (429/rate-limited)

Key semantic contract

Every execution should carry:

• execution_id
• notification_id
• available_until
• channel
• idempotency/dedup token

This lets every downstream stage make correct decisions.

2.3 Important semantics

Exactly-once vs at-least-once

Like many distributed systems:

• Queue delivery / worker execution: typically at-least-once
• User-visible send effect: should be exactly-once logically, enforced by idempotency and state transitions

Expiration semantics

• Promo: hard stop after available_until
• Critical: may also have expiration for certain classes, but usually less aggressive

Priority semantics

• critical and promo must not share the same unconstrained worker/queue pool

3) Data Model + Keys + Consistency

This is where the system becomes real.

We need to model:

• notification intent
• execution/delivery attempts
• campaign expansion
• dedup/idempotency
• expiration

3.1 Core entities

A) Notifications

Represents the logical notification intent.

Fields:

• notification_id
• tenant_id
• type (critical | promotional)
• campaign_id (nullable)
• channel
• template_id
• payload or payload pointer
• created_at
• available_until
• priority_class

For critical direct sends:

• one notification may map closely to one recipient

For promo:

• a campaign-level notification intent may fan out into many executions

B) Campaigns

Represents the promo campaign definition.

Fields:

• campaign_id
• tenant_id
• segment snapshot reference / segment_id
• start_at
• available_until
• channels
• template_id
• status (scheduled, running, paused, cancelled, completed, expired)
• rollout controls
• quotas / rate caps

Why separate campaigns?

Because campaign lifecycle is not the same as per-recipient delivery execution.

C) Executions

This is the most important operational table.

It tracks the per-recipient work item to be sent or attempted.

Fields:

• execution_id
• notification_id
• campaign_id (nullable)
• tenant_id
• recipient_id
• channel
• due_at
• available_until
• status (pending, leased, enqueued, sending, sent, failed, expired, suppressed)
• lease_owner
• lease_until
• version
• attempt_count
• provider
• provider_request_id
• retention_ttl

Why Executions is separate

Because:

• one campaign can create many executions
• retries and delivery state are per-recipient
• this is the main scheduler/work ledger

D) Outbox

Used by the scheduler to guarantee “claimed work becomes enqueued” reliably.

Fields:

• shard_id
• outbox_id
• execution_id
• destination queue/topic
• payload pointer/hash
• state (pending, sent)
• retries
• next_attempt_at

Why outbox matters

It prevents the classic gap:

• scheduler claims item
• crashes before enqueue
• item never gets sent

The outbox closes that gap.

E) DeliveryAttempts (optional)

If you want detailed per-attempt history.

Fields:

• execution_id
• attempt_no
• provider
• provider response
• error class
• latency
• timestamp

When to include

• strong audit requirements
• detailed provider debugging

If interviewer wants simpler MVP, keep attempts folded into Executions.

3.2 Key design / partitioning

At this scale, partitioning matters a lot.

Recommended execution partition key pattern

Use:

• shard ownership + time bucket

Example logical key:

• PK = shard_id:bucket_5m
• SK = due_at#execution_id

This is exactly the kind of pattern that scales scheduler scans well:

• ordered by due time within a shard/time bucket
• avoids giant hot partitions
• supports deterministic shard ownership

Why not one giant table key by campaign?

Because a large campaign can create a hot partition and kill throughput.

Why shard by hash(execution_id) % K?

Because it spreads writes uniformly.

3.3 Segment targeting and fanout model (critical design choice)

This is one of the biggest pitfalls.

Bad approach

Immediately expand a promo campaign targeting 10M users into 10M execution rows at once.

Why bad

• fanout explosion
• queue floods
• DB write spikes
• provider backlog
• campaign may expire before most work can even start

Better approach: controlled expansion

Store in campaign:

• segment snapshot reference (or immutable segment membership snapshot pointer)

Then scheduler:

• pages through membership
• expands in controlled batches
• only while downstream capacity exists
• stops if nearing expiration or queue high-water marks

This is one of the strongest design upgrades.

Principle

The scheduler should meter expansion, not just meter delivery.

That’s very Principal.

3.4 Expiration and TTL (important)

Every promo execution carries:

• available_until

Enforcement points

1. Scheduler
   • do not enqueue work if expired
2. Worker
   • drop on sight if expired before provider call
3. Queue message
   • include available_until in payload
4. DB cleanup
   • TTL can clean old rows eventually, but do not rely on TTL for correctness

If you use DynamoDB TTL, AWS states deletes are best-effort and expired items are typically deleted “within a few days,” so TTL should be treated as cleanup, not correctness logic.

That is a strong, factual point to call out.

3.5 Consistency semantics

Be precise here.

What is at-least-once?

• queue redelivery
• worker execution
• outbox publish retries

What should be exactly-once logically?

• final user-visible delivery effect

How to enforce that

• idempotency key per execution
• state transition guard (only one terminal success)
• provider idempotency if supported
• dedup cache at worker/provider integration layer

Example

If worker sends to provider, crashes before persisting success:

• queue may redeliver
• worker retries
• dedup key prevents duplicate user-visible send (best effort end-to-end)

4) HLD (Components + Updated Diagram)

Now we draw the system.

4.1 Updated HLD diagram

4.2 Component-by-component explanation

1) Notification Ingress API

This is the entry point.

Responsibilities

• validate notification request
• classify as critical or promo
• write initial notification/campaign record
• perform request-level idempotency

Why separate ingress from delivery?

Because ingestion may be fast, while actual delivery can be deferred, retried, throttled, or cancelled.

2) Critical Path Router

Critical notifications need low latency.

Responsibilities

• bypass heavy campaign machinery
• enqueue directly into a fast queue/log
• preserve partitioning for ordering if needed (e.g., by recipient/conversation)

Why separate path?

Because critical traffic cannot wait behind segment expansion and promo backlog.

This is one of the most important architectural separations.

3) Promo Campaign Router

For promo, the system should not immediately explode into all recipient sends.

Responsibilities

• persist campaign definition
• store segment snapshot pointer
• create initial scheduled work state

Why not immediate full fanout?

Because big campaigns can overwhelm storage and queues instantly.

4) Promo Scheduler Fleet

This is the brain of the promotional path.

Responsibilities

• own a deterministic set of shards
• scan only their owned shards for due executions/campaign pages
• expand segment membership in controlled batches
• create/claim executions
• perform expiry checks
• write outbox entries for enqueue

Why shard ownership?

To reduce claim races and simplify distributed coordination.

This is one of the strongest ideas from the Hello Interview discussion.

5) Critical Queue / Log

A fast queueing system (often Kafka-like or another high-throughput log/queue) for critical work.

Why here?

• very high throughput
• low latency
• scalable consumer groups for parallelism

Kafka’s consumer group model is specifically designed so a pool of consumers can divide work across processes/machines for scalability and fault tolerance.

You do not need Kafka specifically, but this is a reasonable design if interviewer asks for examples.

6) Promo Queues

Promo can use a different queueing shape:

• simpler queue semantics
• more elastic worker scaling
• priority isolation per tenant/campaign if needed

If using SQS-like semantics, visibility timeout matters because messages can be re-delivered if not deleted before the timeout expires. AWS documents that SQS messages become visible again when visibility timeout expires, so workers must be idempotent.

7) Channel Worker Pools

Separate worker pools for:

• push
• email
• SMS

Why channel-specific pools?

Because:

• provider APIs differ
• retry semantics differ
• throughput limits differ
• cost and failure patterns differ

This makes scaling and throttling much cleaner.

8) Provider Throttle / Rate Control

A crucial component many candidates omit.

Responsibilities

• enforce per-provider rate limits
• enforce per-tenant and per-campaign quotas
• slow-start ramp-up
• respond to provider 429s by reducing throughput
• protect provider reputation (email warmup, SMS short code limits, etc.)

Why this matters

Even if internal queues are healthy, providers are still bottlenecks.

5) Critical Flows

5.1 Critical notification flow (fast path)

Example: user sends a 1:1 chat message, recipient should get push notification quickly.

Step 1: Ingest

Producer calls ingress API with critical notification request.

Step 2: Validate + idempotency

Ingress:

• validates payload
• checks dedup/request idempotency
• creates notification record

Step 3: Enqueue to critical queue

Notification is enqueued directly onto critical queue/log, partitioned by a stable key such as:

• recipient_id
• or conversation_id (if per-thread ordering matters)

Step 4: Channel worker consumes

Worker:

• reads message
• checks if expired (if critical TTL exists)
• dedups execution
• applies provider throttling
• sends to provider

Step 5: Record terminal state

Worker marks:

• sent
• or retryable failure
• or permanent failure

Why this is fast

It avoids:

• campaign scheduler
• segment expansion
• large DB scans
• promo throttling backlog

That path isolation is the core idea.

5.2 Promotional campaign flow (controlled expansion path)

Example: “recommend this content to all users in segment X before 6 PM.”

Step 1: Campaign creation

Producer creates campaign with:

• campaign_id
• segment snapshot reference
• start_at
• available_until
• channels

Step 2: Campaign stored, not fully exploded

System stores:

• campaign metadata
• reference to segment snapshot or immutable membership pointer

It does not create all recipient sends at once.

Step 3: Promo scheduler wakes up

Scheduler owning relevant shards:

• scans due campaigns/executions
• checks campaign is active and not expired

Step 4: Controlled fanout

Scheduler pages through segment members in small batches

(e.g., 5k–20k users/sec per campaign depending capacity).

For each batch:

• create executions
• write outbox entries
• enforce campaign- and tenant-level expansion budgets
• stop if downstream queues are near high-water mark

This is the biggest design point:

Expansion itself is rate-limited.

Step 5: Outbox publishes to promo queues

Outbox publisher sends execution payloads to promo queue(s).

Step 6: Worker consumes

Worker:

• checks available_until
• if expired, mark expired and drop
• otherwise send to provider via rate-control layer

Step 7: Completion / expiry

Campaign may end as:

• completed
• partially completed (window closed)
• cancelled
• expired

This is realistic and strong to say in interview.

5.3 Expiration flow (promo correctness)

This is a key interview probe.

Rule

A promotional notification must not be sent after available_until.

Enforcement layers

Layer 1: Scheduler

Before enqueue:

• if now > available_until, do not enqueue
• mark execution/campaign as expired

Layer 2: Queue message payload

Carry available_until in the message.

Layer 3: Worker

Before provider call:

• if expired, drop and mark terminal expired

Layer 4: Storage cleanup

TTL can eventually clean old rows, but is not relied on for correctness.

If using SQS-like queues, visibility timeout and redelivery mean a message can reappear later; that makes the worker-side expiry check mandatory. AWS documents that visibility timeout expiration makes a message visible again for another consumer.

That is a great concrete point.

5.4 Retry flow

Transient failure

Examples:

• provider timeout
• provider 5xx
• temporary 429

Action:

• retry with exponential backoff + jitter
• increment attempt count
• respect available_until
• re-check throttles before resending

Permanent failure

Examples:

• invalid push token
• hard bounce
• invalid phone number

Action:

• no retry
• mark failed permanently
• possibly suppress future sends to that destination until refreshed

Important

Retries must be:

• channel/provider aware
• bounded
• expiration aware

6) Scaling (shards, queues, fanout, backpressure)

This is where you show Principal judgment.

6.1 Biggest scaling risks

1. Execution store write pressure
2. Promo fanout explosion
3. Provider bottlenecks
4. Critical path interference by promo
5. Hot campaigns / hot tenants

The design should explicitly solve each.

6.2 Separate critical and promo paths

This is foundational.

Why?

Critical:

• low latency
• smaller per-event fanout
• stronger urgency

Promo:

• large fanout
• can be throttled/cancelled/expired
• backlog is acceptable up to campaign window

Scaling benefit

You can independently scale:

• critical queues and workers
• promo schedulers and workers
• provider-specific pools

This prevents priority inversion.

6.3 Execution store scaling

At 1M notifications/sec, the execution store can become the bottleneck if you treat it as the source of truth for every micro-step.

Better pattern

Use queues as the work to do, and store only meaningful state transitions:

• scheduled / queued
• sent
• failed
• expired
• final

That can significantly reduce write amplification.

If you do persist every execution

Use:

• high shard count
• hash-based partitioning
• time buckets
• narrow scans

This aligns with the shard-and-bucket approach discussed in the Hello Interview thread.

6.4 Promo fanout scaling

This is the hardest part.

Do not do this:

“Campaign starts → generate all recipient executions immediately.”

Do this:

• segment snapshot reference
• controlled expansion by scheduler
• page through recipients
• apply:
  • per-campaign expansion rate
  • per-tenant quotas
  • queue high-water mark backpressure
  • channel/provider capacity checks

Why this works

It turns one giant spike into a managed flow.

Strong phrase

“I treat segment expansion as a first-class resource-consuming operation with its own backpressure.”

That sounds very strong.

6.5 Scheduler scaling and coordination

Use deterministic shard ownership

Assign:

• shard_id = hash(execution_id or campaign page token) % K

Each scheduler instance owns a disjoint shard range.

Benefits

• avoids most claim races
• easier scans
• simpler failover
• predictable work ownership

If a scheduler dies:

• another instance takes that shard range
• leases and outbox replay handle in-flight safety

This is one of the best ideas to include.

6.6 Provider-aware throttling (internal + external backpressure)

Providers are external bottlenecks.

Use hierarchical token buckets:

• global per-channel
• per-provider
• per-tenant
• per-campaign
• optionally per-domain/carrier/country

Why hierarchical?

Because a Gmail domain throttle problem is different from an SMS aggregator cap.

Slow-start

Ramp up gradually for big campaigns:

• don’t immediately hit full rate
• increase as success rate remains stable

This is realistic and advanced.

6.7 Queue isolation / priority isolation

Use separate queues or queue classes for:

• critical vs promo
• optionally per-tenant or per-campaign for large promo sends

Why?

To avoid:

• one large campaign starving all other promo
• promo interfering with critical
• unhealthy retry traffic poisoning everything

7) Failure Modes / Resilience

7.1 Scheduler crash

Risk

Scheduler claims work, then crashes before enqueueing.

Solution

Use claim + outbox pattern:

• claim execution (or campaign page)
• write outbox intent
• outbox publisher enqueues
• if scheduler dies, outbox replay recovers unsent intents

This is a very strong design point.

7.2 Queue redelivery / duplicate processing

Risk

Message reappears due to consumer crash or visibility timeout expiry.

SQS-like systems explicitly rely on visibility timeout to hide a message during processing, and if the worker does not delete it before the timeout, it can be delivered again.

Solution

• worker idempotency
• execution status guard
• provider dedup token if supported
• dedup cache keyed by execution_id

7.3 Promo backlog grows beyond campaign window

Risk

Queue delay grows; many promo messages will become stale before send.

Response

• continuously monitor oldest message age / projected drain time
• if projected delivery misses available_until, stop further expansion
• auto-cancel remaining work
• mark campaign partially completed / expired remainder

This is very practical and often overlooked.

7.4 Provider outage / 429 storm

Risk

Provider rejects or slows requests, retries can explode.

Response

• classify errors
• reduce token bucket rate on throttled provider
• circuit breaker per provider/region
• route to backup provider if possible
• stop expansion of affected campaigns if downstream can’t keep up

Important

Retries should not continue at full speed into a failing provider.

7.5 Execution store outage

Risk

Cannot write status transitions or scheduler state.

Response

• critical path may continue with bounded in-memory buffering only if safe and short-lived
• otherwise shed new ingest for affected classes
• do not pretend success without state
• keep idempotency boundary clear

Safer posture

It’s usually better to temporarily stop non-critical campaign expansion than to create unknown delivery state.

8) Tradeoffs / Why Decisions

This is where you sound Principal instead of just “correct.”

8.1 Why split critical and promo?

Because they have fundamentally different SLOs and failure tolerance:

• critical = low latency, low tolerance for delay
• promo = high fanout, delay-tolerant, expiration-sensitive

A single shared pipeline will create priority inversion.

8.2 Why not immediate promo full fanout?

Because it causes:

• DB spikes
• queue spikes
• provider overload
• waste if campaign expires mid-way

Controlled expansion is more stable and more cost-efficient.

8.3 Why use scheduler shard ownership?

Because it reduces:

• coordination contention
• duplicate claims
• race complexity

It gives deterministic ownership and scales horizontally.

8.4 Why outbox?

Because “claim then enqueue” is not atomic across DB + queue.

Outbox gives reliable handoff and closes the “claimed but never enqueued” gap.

8.5 Why provider throttling as a first-class component?

Because queues alone do not solve provider limits.

Without provider-aware throttling:

• retries amplify
• provider reputation can suffer
• campaigns can collapse into failure loops

8.6 Why not rely on TTL for expiration correctness?

Because TTL cleanup is delayed/best-effort in many datastores.

For example, DynamoDB TTL deletes are best-effort and typically occur within days, so correctness must be enforced at read/dispatch/worker time, not by background deletion alone.

That’s a strong factual tradeoff.

9) Observability (SLIs, metrics, tracing, alerts)

A notification system needs strong observability because “it didn’t send” can be hard to debug.

9.1 IDs to carry

Every event should include:

• request_id
• notification_id
• campaign_id
• execution_id
• tenant_id
• channel
• provider
• provider_request_id
• shard_id

That makes end-to-end tracing possible.

9.2 Core SLIs / metrics

Critical path

• enqueue-to-send latency
• queue age
• send success rate
• duplicate suppression rate

Promo path

• expansion rate per campaign
• queue age / projected drain time
• expired-before-send count
• partial campaign completion rate

Provider metrics

• success / 4xx / 5xx / 429 rate
• throttle token utilization
• circuit breaker state
• latency per provider

Scheduler metrics

• shard lag
• outbox backlog
• claim failures
• lease reclaims
• scan latency

Storage metrics

• writes/sec
• hot shard distribution
• TTL cleanup lag (if relevant)
• status transition failures

9.3 Alerts

Important alerts include:

• critical queue age breach
• promo projected drain time exceeds campaign window
• provider 429 spike
• provider circuit breaker open
• expired promo sends attempted (should be near zero)
• outbox backlog spike
• shard ownership lag / unowned shard
• duplicate send anomaly

9.4 Tracing

A trace should show:

• ingest accepted
• path chosen (critical/promo)
• scheduler claim / outbox publish
• queue enqueue
• worker receive
• provider send attempt
• final terminal state

That is how you debug “why didn’t user X get notification Y?”

Recommended final design (what I would present)

High-level design choice

I would design a dual-path notification system:

Critical path

• direct ingest → fast queue/log → channel worker → provider
• optimized for low latency and isolation

Promotional path

• ingest campaign → store segment snapshot ref → shard-owned scheduler → controlled fanout → promo queue → worker → provider
• optimized for bounded expansion, expiration correctness, and provider-aware throttling

Key correctness mechanisms

• idempotent ingest
• per-execution dedup
• outbox for scheduler enqueue safety
• expiry check at scheduler and worker
• provider-aware retry classification

Key scaling mechanisms

• separate queues and workers by priority
• shard-owned scheduler scans
• controlled campaign expansion
• hierarchical token buckets for provider throttling
• bounded retries and backpressure

That’s a strong, production-grade story.

30-second interview summary

“I’d split the system into two isolated paths: a low-latency critical path that goes straight from ingest into a fast queue and channel workers, and a promotional path that stores campaign intent, then uses a shard-owned scheduler to expand recipients in controlled batches. The promo scheduler uses claim-plus-outbox to enqueue safely, and both the scheduler and workers enforce available_until so expired promotions are never sent. I’d isolate channel workers, apply provider-aware token buckets and circuit breakers, and use explicit execution state plus dedup keys to get at-least-once processing internally but exactly-once logical delivery effects.”

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.