Celery
Task Queuing
Task Routing
Python Programming
Distributed Task Processing

How to route a chain of tasks to a specific queue in celery?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

Routing a Celery chain to a specific queue is really a routing-options problem, not a chain problem by itself. The safest approach is to assign the queue explicitly to each signature in the chain or to route the task names through Celery configuration, rather than assuming that one top-level chain option will magically propagate the way you want.

How Celery Routing Works

Celery workers listen to queues, and tasks are published to those queues with routing metadata. A chain is just a sequence of signatures, so each task in that sequence can carry its own execution options.

That means queue routing can be applied in two main ways:

  • configure routing centrally with task_routes
  • set the queue explicitly on each signature with .set(queue=...)

For predictable behavior, especially in composed workflows, explicit per-signature routing is often easiest to reason about.

Explicit Queue Routing Per Task

Here is a small example:

python
1from celery import Celery, chain
2
3app = Celery("demo", broker="redis://localhost:6379/0")
4
5
6@app.task
7def add(a, b):
8    return a + b
9
10
11@app.task
12def multiply(value, factor):
13    return value * factor
14
15
16workflow = chain(
17    add.s(2, 3).set(queue="priority"),
18    multiply.s(10).set(queue="priority"),
19)
20
21result = workflow.apply_async()
22print(result.get(timeout=10))

In this version, both tasks are explicitly sent to the priority queue.

That is more reliable than relying on an ambiguous chain-level shortcut because each signature carries the routing choice directly.

Central Routing With task_routes

If certain tasks should always go to a particular queue, configure routing globally instead of repeating .set(queue=...) everywhere.

python
1from celery import Celery
2
3app = Celery("demo", broker="redis://localhost:6379/0")
4
5app.conf.task_routes = {
6    "demo.add": {"queue": "priority"},
7    "demo.multiply": {"queue": "priority"},
8}

Then your chain can stay simpler:

python
1workflow = chain(
2    add.s(2, 3),
3    multiply.s(10),
4)

This is often the better long-term approach when routing is a stable policy rather than a one-off workflow decision.

Start a Worker for the Queue

Routing only matters if a worker is actually listening to that queue.

bash
celery -A demo worker -Q priority --loglevel=INFO

If no worker consumes the queue, the tasks may publish successfully but sit unprocessed.

This is one of the most common reasons routing “looks broken.” The publishing side can be correct while the worker topology is incomplete.

Mixed-Queue Chains

A chain does not need every task to use the same queue. Sometimes one task should run on a CPU-heavy queue and the next on a default queue.

python
1workflow = chain(
2    add.s(2, 3).set(queue="cpu"),
3    multiply.s(10).set(queue="default"),
4)

That is another reason to think in per-signature routing. A chain is a workflow, not necessarily a single-queue unit.

Routing and Result Injection

Queue routing does not change how Celery passes results through a chain. The previous task result is still injected into the next task unless you use immutable signatures such as .si(...).

So these are separate concerns:

  • queue routing decides where a task runs
  • chain semantics decide how arguments flow between tasks

Keep those concepts separate when debugging.

When to Prefer Configuration Over Inline .set

Inline .set(queue=...) is great when the workflow itself decides the queue. task_routes is better when the queue is part of the task’s normal identity.

Good candidates for central routing:

  • always-heavy image processing tasks
  • always-low-priority cleanup jobs
  • dedicated integration queues per subsystem

Good candidates for inline routing:

  • one-off workflows
  • experimental routing
  • dynamic queue choice at runtime

Use the style that matches whether routing is policy or situation.

Common Pitfalls

The most common pitfall is assuming a single top-level chain setting always propagates exactly as intended to every task. The safer answer is to route each signature explicitly or define task_routes.

Another mistake is routing tasks to a queue that no worker is consuming.

A third issue is mixing routing concerns with argument-flow concerns. Queue placement and result injection are different parts of Celery behavior.

Finally, teams sometimes hardcode queue names everywhere instead of centralizing stable routing policy, which makes later maintenance harder.

Summary

  • The safest way to route a Celery chain is to set the queue on each task signature or define task routing centrally.
  • Queue routing controls where tasks run, not how chain arguments are passed.
  • Start workers with the correct -Q setting so routed tasks actually get consumed.
  • Use per-signature routing for dynamic workflows and task_routes for stable policy.
  • Treat a chain as a workflow of routed tasks rather than assuming it is a single queue object.

Course illustration
Course illustration

All Rights Reserved.