Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation but supports scheduling as well. The execution units, called tasks, are executed concurrently on one or more worker nodes using multiprocessing, Eventlet, or gevent. Celery is used in production systems to process millions of tasks a day.

How Celery Works with Django and RabbitMQ

Celery can be integrated into Django applications to handle background tasks. RabbitMQ, often used as the messaging system, acts as the message broker for Celery. A message broker is a component that translates a message from the messaging protocol of the sender to the messaging protocol of the receiver. Celery communicates via messages, usually using a broker to mediate between clients and workers.

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Thanks to Django's scalability, Celery works well for handling background tasks in Django applications.

RabbitMQ is one of the most popular open source message brokers. It supports multiple messaging protocols. RabbitMQ can handle large volumes of messages and has excellent support for clustering and reliability.

Setting Up Celery with Django and RabbitMQ

Step 1: Install Celery

Install Celery and its dependencies with:

pip install celery

Step 2: Configure Celery in Django

Add a new module called celery.py in your Django project’s main module.

1from __future__ import absolute_import, unicode_literals
2import os
3from celery import Celery
4
5# set the default Django settings module for the 'celery' program.
6os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')
7
8app = Celery('your_project')
9
10# Using a string here means the worker doesn't have to serialize
11# the configuration object to child processes.
12app.config_from_object('django.conf:settings', namespace='CELERY')
13
14# Load task modules from all registered Django apps.
15app.autodiscover_tasks()

Step 3: Set Up the RabbitMQ Broker

Ensure RabbitMQ is installed and running on your machine or on your designated server. Configure the BROKER_URL in your Django settings:

CELERY_BROKER_URL = 'amqp://myuser:mypassword@localhost/myvhost'

Example of a Simple Task

Create a tasks.py file in one of your Django apps and define a task:

1from celery import shared_task
2
3@shared_task
4def add(x, y):
5    return x + y

This task can be called in the background in various parts of your Django application:

from myapp.tasks import add
add.delay(4, 4)

Scheduling Tasks with Celery Beat

Celery Beat is a scheduler that kicks off tasks at regular intervals, which are then executed by available worker nodes in the cluster. Tasks can be scheduled to execute periodically or at a specific time.

Simple Periodic Tasks

To use Celery Beat, you need to setup a periodic tasks schedule:

1from celery import Celery
2from celery.schedules import crontab
3
4app = Celery()
5
6app.conf.beat_schedule = {
7    'add-every-30-seconds': {
8        'task': 'myapp.tasks.add',
9        'schedule': 30.0,
10        'args': (16, 16)
11    },
12    'add-every-minute-crontab': {
13        'task': 'myapp.tasks.add',
14        'schedule': crontab(),
15        'args': (16, 16)
16    },
17}

Key Concepts Table

By implementing Celery with Django and RabbitMQ, developers can effectively manage background tasks and periodic tasks to enhance web application performance. This combination not only leverages Django's rapid development features but also ensures reliable message queuing through RabbitMQ and scalable task management with Celery.