C#
Celery
Task Queue
Programming
Interoperability

How can I queue a task to Celery from C#?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

Yes, a C# application can queue work for Celery, but the important detail is that you are not calling Python code directly. You are publishing a Celery task message through the broker using the task name, serializer, headers, routing, and body format that the Celery worker expects.

The Simplest Architecture

Celery workers consume messages from a broker such as RabbitMQ or Redis. From C#, the cleanest designs are usually:

  1. call a small Python or HTTP service that uses Celery normally
  2. publish Celery-compatible broker messages directly from C#

The first option is easier to maintain. The second option is possible, but only if you implement the Celery task protocol correctly.

Why Direct Broker Publishing Is Tricky

Celery is language-agnostic at the protocol level, and the official docs explicitly note that the protocol can be implemented in other languages. But that does not mean "publish any JSON to RabbitMQ." A Celery worker expects:

  • a task name
  • a task id
  • headers that identify the protocol and task
  • a body that matches the configured serializer and protocol version
  • routing that sends the message to a queue the worker is actually consuming

If any of those do not match the worker configuration, the task will not run.

If you control both systems, the most robust approach is often to expose a tiny Python endpoint that calls send_task or apply_async.

python
1from celery import Celery
2
3app = Celery("demo", broker="amqp://guest:guest@localhost//")
4
5def queue_add(x, y):
6    return app.send_task("proj.tasks.add", args=[x, y])

Then your C# application can call that service over HTTP or gRPC. This keeps Celery protocol details inside Python, where Celery already knows how to do the right thing.

Direct RabbitMQ Publishing From C#

If you do want direct publishing, RabbitMQ is a practical broker because C# has strong AMQP client libraries. The message must match Celery's task protocol. For protocol version 2, Celery documents headers such as task, id, and lang, and a body shaped like (args, kwargs, embed) for Python-style JSON messages.

Here is a minimal RabbitMQ example that publishes a JSON task message compatible with a default-style Celery worker setup.

csharp
1using System;
2using System.Collections.Generic;
3using System.Text;
4using System.Text.Json;
5using RabbitMQ.Client;
6
7var factory = new ConnectionFactory
8{
9    HostName = "localhost",
10    UserName = "guest",
11    Password = "guest"
12};
13
14using var connection = factory.CreateConnection();
15using var channel = connection.CreateModel();
16
17var taskId = Guid.NewGuid().ToString();
18var body = JsonSerializer.Serialize(new object[]
19{
20    new object[] { 2, 2 },
21    new Dictionary<string, object>(),
22    null
23});
24
25var properties = channel.CreateBasicProperties();
26properties.ContentType = "application/json";
27properties.ContentEncoding = "utf-8";
28properties.CorrelationId = taskId;
29properties.Headers = new Dictionary<string, object>
30{
31    ["lang"] = "py",
32    ["task"] = "proj.tasks.add",
33    ["id"] = taskId,
34    ["argsrepr"] = "(2, 2)",
35    ["kwargsrepr"] = "{}"
36};
37
38channel.BasicPublish(
39    exchange: "",
40    routingKey: "celery",
41    basicProperties: properties,
42    body: Encoding.UTF8.GetBytes(body)
43);

This example assumes the worker is consuming the default queue named celery. If your Celery app uses custom queues, exchanges, routing keys, or serializers, the C# publisher must match those settings exactly.

Match the Serializer and Task Name

The easiest cross-language choice is JSON. Avoid pickle because it is Python-specific and inappropriate for a C# producer. Also make sure the task header matches the fully qualified Celery task name registered by the worker, such as proj.tasks.add.

If the worker expects a different serializer or custom routing, mirror that configuration in the C# publisher rather than guessing.

Result Handling

Queueing a task is only half the problem. If you also need results, decide whether you want:

  • fire-and-forget task submission
  • polling through a result backend
  • a separate application-level completion callback

Cross-language result handling can be more work than task submission itself, so define that contract early.

Common Pitfalls

The biggest pitfall is assuming RabbitMQ alone is enough. RabbitMQ only transports the message. Celery still requires a specific task protocol on top of the broker.

Another issue is sending JSON that does not match the serializer or body format the worker expects. A message can arrive in the queue and still fail to execute.

Developers also forget about routing. Publishing to the wrong queue or exchange makes the task invisible to the worker even when the message format is correct.

Finally, if the system will evolve over time, direct protocol publishing from C# creates a maintenance burden. A thin Python producer service is often safer.

Summary

  • A C# app can queue Celery tasks by publishing Celery-compatible messages to the broker.
  • The recommended low-friction option is often a thin Python service that calls Celery normally.
  • If you publish directly, match Celery's task protocol, task name, serializer, and routing exactly.
  • JSON is the safest serializer for cross-language interoperability.
  • Broker interoperability is possible, but protocol compatibility is what actually makes Celery work.

Course illustration
Course illustration

All Rights Reserved.