Design a Task Scheduler

You schedule a daily report to run at midnight. Simple, right? Now multiply that by 100K users who all scheduled their daily tasks for midnight too. The system must fire 100K tasks within seconds of 00:00:00, ensure each runs exactly once even with multiple dispatcher instances competing, retry failures without duplicating side effects, and handle a worker crashing mid-execution. The real challenge is not running a task. It is guaranteeing that every task runs once, on time, despite failures.

Functional

One-time task scheduling: Users schedule a task to execute at a specific future time. The system accepts a target (HTTP callback URL or queue message), a timestamp, a priority level, and an optional idempotency key. The task fires once and reports success or failure.

The midnight spike reveals the architecture. If 40% of recurring tasks are scheduled for midnight, a system handling 1000 tasks/sec steady-state faces a burst of 100K tasks in a single second. This burst capacity requirement (not the steady-state throughput) drives the decision to use Redis sorted sets instead of polling PostgreSQL directly.

Steady-State

Scheduling rate: 1000 tasks per second created via the API. Each task carries roughly 2KB of metadata: UUID (36 bytes), schedule timestamp, target configuration as JSON (URL, headers, payload, typically 1KB), retry policy (max attempts, backoff strategy), priority (integer 1-10), and idempotency key (36 bytes).

Two types of tasks need two separate endpoints. A one-time task has a run_at timestamp and executes once. A recurring task has an interval and generates concrete task instances forever. Conflating them into one endpoint creates a leaky abstraction. The scheduling logic, cancellation semantics, and status queries are fundamentally different.

Create One-Time Task

POST /v1/tasks

Two storage tiers matched to two access patterns. PostgreSQL for durable task metadata. The source of truth that survives any crash. Redis sorted set for near-future dispatch. Sub-second pop of the next due task. PostgreSQL answers "what tasks exist?" Redis answers "what tasks are due right now?"

PostgreSQL, tasks Table

1CREATE TABLE tasks (
2id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
3recurring_task_id UUID REFERENCES recurring_tasks(id),
4status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
5run_at TIMESTAMP NOT NULL,
6started_at TIMESTAMP,
7completed_at TIMESTAMP,
8target_type VARCHAR(20) NOT NULL,
9target_config JSONB NOT NULL,
10attempts INT DEFAULT 0,
11max_attempts INT DEFAULT 3,
12priority INT DEFAULT 5,
13idempotency_key VARCHAR(255) UNIQUE,
14created_at TIMESTAMP DEFAULT now()
15);
16
17CREATE INDEX idx_tasks_dispatch ON tasks (status, run_at)
18WHERE status = 'PENDING';

Five components, one key separation. The scheduling path and the execution path are decoupled by a queue. A task enters through the API Service, gets durably stored in PostgreSQL, pre-loaded into Redis by the Scheduler, popped and claimed by the Dispatcher, and executed by a Worker. Each component scales independently.

API Service: Stateless. Validates task payloads (checks that run_at is in the future, target_config has a valid URL, priority is 1-10), writes to PostgreSQL with status=PENDING, and pushes near-future tasks (due within 5 minutes) to Redis via ZADD. Horizontally scalable behind a load balancer. Does not participate in task execution, its only job is to durably accept tasks and confirm acceptance to the client.

Two flows define this system. Task creation and execution. The happy path from API to worker. Recurring task generation. The background loop that turns a template into concrete instances.

Task Creation and Execution Flow

What happens when two dispatchers pop the same task from Redis? This is the real interview question. On a single Redis instance, ZPOPMIN is atomic. Once popped, the element is gone and no other client can pop it. But during a Redis failover (replica promoted to primary before replication caught up), a task that was popped from the old primary might still exist on the new primary. Two dispatchers could then hold the same task_id. Without a second guard, the task executes twice. Charging a customer twice, sending two emails, or corrupting data.

Atomic Task Claiming (GATE)

Three cascading decisions shape this system, where to queue tasks (DB-only vs DB+Redis), what execution guarantee to provide (at-least-once vs exactly-once), and how to handle recurring schedules (simple intervals vs full cron).

DB-Only Polling vs DB+Redis Two-Tier

Every component will fail. The question is whether a failure causes tasks to be lost, duplicated, or delayed, and which is acceptable. This system is designed so that failures cause delay (higher latency) but never loss (tasks are in PostgreSQL) and rarely duplication (atomic claiming + idempotency keys).

Redis Failure

The sorted set is lost. Dispatchers cannot pop tasks. Fallback: dispatchers detect the Redis connection failure (connection timeout or ECONNREFUSED) and switch to direct DB polling with SELECT FOR UPDATE SKIP LOCKED. Latency increases from sub-second to 1-5 seconds (polling interval). No tasks are lost. They are all in PostgreSQL with status=PENDING.

The system as designed handles reliable task scheduling at scale with sub-second accuracy. The next frontier is richer scheduling expressions, multi-region execution, and platform capabilities. Each improvement below can be layered on top of the existing architecture without redesigning the core dispatch path.

Full Cron with Timezone Support

Extend recurring tasks to accept cron expressions (e.g., "0 9 * * MON-FRI" for weekday mornings) with per-task timezone configuration. Handle DST transitions explicitly. Spring-forward skips the non-existent hour (a task scheduled for 2:30 AM does not run), fall-back runs once at the first occurrence of the ambiguous hour (not twice). Requires a calendar-aware parsing library (like cronstrue or node-cron), a timezone column on the recurring_tasks table, and validation logic to reject expressions that would fire more than once per minute. The core dispatch path (Redis pop, atomic claim, worker execution) remains unchanged, only the scheduler's task-generation logic becomes more complex.