Introduction

A Django application can consume Kafka messages, but the consumer should usually run outside the normal request-response path. The cleanest pattern is to treat Kafka consumption as a separate long-running worker that uses Django's settings and models, rather than trying to read Kafka inside view functions or middleware.

Why a Web Request Is the Wrong Place for a Consumer

A Kafka consumer is normally a continuous process that:

• joins a consumer group
• polls for new records
• processes messages in a loop
• commits offsets carefully

That does not fit the lifecycle of an HTTP request. A Django view should respond and finish. A Kafka consumer should usually stay alive for the lifetime of a worker process.

So the common answer is: integrate with Django, but run the consumer as a management command or separate service.

A Simple Django Management Command

A good starting point is a custom management command.

1from django.core.management.base import BaseCommand
2from confluent_kafka import Consumer
3
4
5class Command(BaseCommand):
6    help = "Consume Kafka messages"
7
8    def handle(self, *args, **options):
9        consumer = Consumer({
10            "bootstrap.servers": "localhost:9092",
11            "group.id": "django-workers",
12            "auto.offset.reset": "earliest",
13        })
14
15        consumer.subscribe(["orders"])
16
17        try:
18            while True:
19                msg = consumer.poll(1.0)
20                if msg is None:
21                    continue
22                if msg.error():
23                    self.stderr.write(str(msg.error()))
24                    continue
25
26                value = msg.value().decode("utf-8")
27                self.stdout.write(f"Received: {value}")
28        finally:
29            consumer.close()

Run it with:

python manage.py consume_kafka

This keeps the consumer inside the Django project while still treating it as a background worker.

Processing Messages with Django Models

Once the worker is running, you can use Django ORM logic inside the command.

1import json
2from myapp.models import OrderEvent
3
4payload = json.loads(value)
5OrderEvent.objects.create(
6    external_id=payload["id"],
7    status=payload["status"],
8)

That is one of the main reasons to use a management command: you can reuse Django configuration, models, and settings instead of creating a separate unrelated script.

Commit Offsets Carefully

Real consumers need a deliberate strategy for offset commits. If you commit too early, you can lose messages after a crash. If you never commit, you can reprocess the same messages repeatedly.

A common safe rule is:

1. read the message
2. process it successfully
3. commit the offset

For example:

1consumer = Consumer({
2    "bootstrap.servers": "localhost:9092",
3    "group.id": "django-workers",
4    "auto.offset.reset": "earliest",
5    "enable.auto.commit": False,
6})

Then after successful processing:

consumer.commit(message=msg)

That gives you explicit control over when a message is considered done.

Run It as a Separate Service

Even though the command lives in Django, it should usually be supervised like a worker process, not like a web endpoint.

Typical deployment patterns include:

• systemd service
• Supervisor
• Docker container separate from the web container
• process manager entries in a platform deployment

That separation keeps web traffic concerns and stream-consumption concerns from stepping on each other.

Think About Idempotency

Kafka consumers may see retries or redeliveries depending on failure timing. If a message can be processed more than once, your database write path should ideally be idempotent.

Examples:

• enforce a unique external event ID
• use upserts instead of blind inserts
• check whether the work was already applied

This matters just as much as the consumer loop itself.

Common Pitfalls

The biggest mistake is trying to consume Kafka messages directly inside Django views or startup hooks in the web server process. That creates lifecycle and scalability problems quickly.

Another issue is forgetting offset strategy. If you let the consumer commit automatically without understanding the timing, you may lose messages after partial processing failures.

Developers also underestimate operational concerns. A consumer should be restarted on failure, logged clearly, and monitored like any other worker.

Finally, do not forget database-side deduplication or idempotency rules. Kafka delivery and application-side write safety need to be designed together.

Summary

• A Django app can consume Kafka, but the consumer should usually run as a separate worker process.
• A custom Django management command is a practical integration pattern.
• Process the message first and commit offsets deliberately.
• Reuse Django ORM and settings inside the worker rather than inside HTTP request handlers.
• Design for retries and duplicate delivery from the start.