Requirements

Functional Requirements:

• A user can schedule (Create and RUD) a task
• A user can configure intervals for recurring tasks
• A user can be notified of task execution automatically

Non-Functional Requirements:

• The system should efficiently manage and execute thousands of tasks

• High availability and reliability - if a scheduler dies, the tasks must survive and be rebalanced across instances.

• Any errors in execution should be logged and reported immediately

• Execution rate < 50ms - between scheduled time and execution with minimal jitter

• High priority tasks get executed before low priority tasks

• Idempotency - if a worker retries a failed task, you need to prevent doubled execution.

Capacity Estimation

• 1M DAU and ~10M registered users
• ~2 tasks/day -> 2M tasks,
• ~23 scheduling API requests/sec, spiking to ~1-10K/sec peak
• ~100-1000 tasks/sec executed at peak
• Big spikes can be upwards of 10-50x the average -> system must queue and buffer bursts
• ~1-2KB per task -> 2M/day -> ~3-4GB/day -> ~1-1.5 TB/year of task metadata
• Execution logs grow even faster: ~0.5KB per run -> 2M exec/day -> ~1 GB/day -> ~350 GB/year.
• Planning for retention/archival policy is crucial (90 day hot, the rest archived)

API Design

POST /tasks

Request: { "type": "send_email", "payload": {...}, "run_at": "2026-06-01T09:00:00Z" }

Response: { "task_id": "t_123", "status": "PENDING" }

POST /recurring-tasks

Request: { "type": "daily_report", "payload": {...}, "interval": "24h", "start_at": "...", "max_runs": 365 }

Response: { "recurring_task_id": "rt_456", "next_run_at": "..." }

GET /tasks/{id} - fetch status, timestamps, error details, attempt count

POST /tasks/{id}/cancel - cancel a pending tasks (plus pause/resume for recurring)

GET /health or /metrics - health + queue depth, execution rate, failure rate

High-Level Design

User calls the API -> validates and writes the task to PG DB (source of truth) -> a scheduler scans the DB for tasks due in the future and pre-loads them into Redis -> dispatcher claims due tasks from Redis and hands to workers -> workers execute and write success/failed back -> the notification service fires on completion or failure

PostgreSQL - durable

Redis - fast dispatch - instant polling

Two separate queues - DB holds ALL tasks, Redis holds the soon-to-run ones

Sharding sync for scheduler instances on task_id for scalability

Database Design

A PostgreSQL DB for all tasks stored

• Tasks
  • task_id: bigint (pk)
  • type: varchar
  • payload: jsonb
  • run_at: timestamp
  • status: varchar
  • attempts: int
  • isPriority: bool
  • recurring_id: bigint (fk)
• Recurring_Tasks
  • recurring_id: bigint (pk)
  • type: varchar
  • payload: jsonb
  • interval: interval
  • next_run_at: timestamp
  • last_run_at: timestamp
  • max_runs: int
• Execution_Logs
  • log_id: bigint (pk)
  • task_id: bigint (fk)
  • started_at: timestamp
  • finished_at: timestamp
  • result: varchar
  • error: text

Redis for soon to execute tasks: ZSET where score = run_at, holding tasks due int he next few minutes.

Detailed Component Design

The Dispatcher - atomic task claiming (the gate)

multiple instances polling Redis - two of them could grab the same task and both execute it -> danger of duplication -> the fix is atomic claim -> status='PENDING' is the guard -> dispatcher 1 claims first, dispatcher 2 will skip -> no locking schemes or race condition protection needed

The Scheduler - near-future preloading + cold-start recovery

the scheduler scans PG DB for tasks due inn the next few minutes and pushes to Redis ZSET (score = run_at) -> Dispatchers pop tasks where score <= now()

• Cold Redis after a restart - if Redis dies, the ZSET is empty and near-future tasks could be silently missed -> on startup, the scheduler re-scans the DB for tasks with run_at within the window and re-loads them - "cache warm-ups."
• Sub-100ms jitter: dispatchers can additionally pre-load the next 1-2 seconds of tasks into an in-memory min-heap and fire them iwth a high res time, instead of relying on polling intervals.

Recurring task generation - avoid the race

2 schedulers both scan `recurring_tasks` for `next_run_at` <= now() -> both create concrete task for the same interval -> duplicate run: Fix with compare-and-swap on `next_run_at

Catch-up policy

• RUN_ALL_MISSED - backfill every missed interval (accounting/audit jobs)
• RUN_ONE_NOW - run once immediately (status refreshes)
• SKIP_MISSED - advance to next future slot (non-critical checks)