Introduction

Python RQ workers are long-running processes, so shutdown behavior must be intentional if you want clean deployments and no lost work. The goal is usually a graceful stop: finish the current job, then exit. Dynamic worker management becomes reliable when you combine OS signals, process supervision, and queue-aware orchestration.

Understand Warm vs Cold Shutdown

RQ workers react differently to signals. SIGINT and SIGTERM can request graceful behavior, while repeated interrupts may force faster termination depending on worker state and version.

1# worker.py
2import os
3from redis import Redis
4from rq import Connection, Worker, Queue
5
6listen = ["default"]
7redis_conn = Redis(host="localhost", port=6379)
8
9if __name__ == "__main__":
10    with Connection(redis_conn):
11        worker = Worker(map(Queue, listen), name=f"worker-{os.getpid()}")
12        worker.work(with_scheduler=True)

Start this worker from a process manager so signal delivery is explicit and observable.

Launch and Stop Workers Dynamically

When workers are started by your own controller, keep process IDs and send signals programmatically.

1import os
2import signal
3import subprocess
4import time
5
6
7def start_worker() -> subprocess.Popen:
8    return subprocess.Popen(["python", "worker.py"])
9
10
11def stop_worker_gracefully(proc: subprocess.Popen, timeout_sec: int = 30) -> None:
12    proc.send_signal(signal.SIGTERM)
13    start = time.time()
14
15    while proc.poll() is None and (time.time() - start) < timeout_sec:
16        time.sleep(0.5)
17
18    if proc.poll() is None:
19        proc.kill()
20
21
22if __name__ == "__main__":
23    worker = start_worker()
24    time.sleep(5)
25    stop_worker_gracefully(worker)

This pattern gives a warm shutdown window and a fallback hard stop if a job hangs.

Coordinate with Queue State

Blindly stopping workers can increase latency if the queue still has high backlog. A better approach is drain-aware scaling:

• stop accepting new traffic for the producer path
• observe queue depth
• shrink worker count gradually

1from redis import Redis
2from rq import Queue
3
4redis_conn = Redis(host="localhost", port=6379)
5queue = Queue("default", connection=redis_conn)
6
7print(f"pending jobs: {queue.count}")

When depth is low, retire workers one by one to reduce disruption.

Use a Supervisor for Real Deployments

In production, use systemd, Supervisor, or container orchestration so workers restart only when desired and signals propagate correctly.

For Kubernetes, send SIGTERM, give a terminationGracePeriodSeconds, and keep job durations below that threshold when possible. Add readiness controls on producers during rollout to avoid creating new backlog while workers are draining.

Draining in Container Platforms

If workers run in pods, scale down gradually instead of deleting all replicas at once. A common sequence is:

• pause job producers
• wait for queue depth to drop under a threshold
• reduce worker deployment replicas stepwise
• resume producers after health checks

This prevents sudden queue spikes and keeps operational risk low during deploy windows.

Handle Long Jobs Safely

If jobs can run for several minutes, include checkpointing or idempotency logic so retries after forced termination do not corrupt state. Keep jobs side-effect-aware and persist progress where practical.

1def process_item(item_id: str) -> None:
2    # Example idempotent guard
3    if already_processed(item_id):
4        return
5
6    do_work(item_id)
7    mark_processed(item_id)

Graceful shutdown is much easier when job handlers can be retried safely.

Common Pitfalls

• Sending SIGKILL immediately can interrupt active jobs and leave partial side effects.
• Managing workers without a supervisor often leads to orphan processes.
• Scaling down all workers at once can create abrupt queue latency spikes.
• Ignoring idempotency makes forced shutdowns risky for billing, emails, and external writes.
• Not monitoring queue depth during shutdown causes poor operational decisions.

Summary

• Prefer graceful stop signals first, then hard kill only after timeout.
• Track worker processes explicitly when doing dynamic scale operations.
• Use queue depth to decide when and how many workers to retire.
• Rely on process supervisors for reliable lifecycle management.
• Design jobs to be idempotent so shutdown and retry paths are safe.