Introduction

If you need only one instance of a long-running Celery task to run at a time, the important requirement is not "only one worker thread touches this code." The real requirement is a distributed lock that works across all workers and all machines.

A process-local flag or a Python global variable is not enough. Celery workers are separate processes, and in real deployments they may run on several hosts.

Decide What "Only One Instance" Means

There are two common meanings:

• only one task of this kind may run globally
• only one task with the same argument set may run at a time

That distinction affects the lock key. A task that imports one customer account at a time may use a per-customer lock, while a nightly global sync may use a single global key.

A Simple Distributed Lock With Redis

One practical pattern is to use Redis and take a lock before doing the work:

1from celery import shared_task
2from django.core.cache import cache
3import time
4
5
6@shared_task
7def run_report():
8    lock_key = "lock:run_report"
9    lock_timeout_seconds = 60 * 10
10
11    acquired = cache.add(lock_key, "1", timeout=lock_timeout_seconds)
12    if not acquired:
13        return "already-running"
14
15    try:
16        time.sleep(5)  # stand-in for real long-running work
17        return "completed"
18    finally:
19        cache.delete(lock_key)

cache.add(...) is useful because it succeeds only if the key does not already exist. That gives you a basic mutual-exclusion mechanism across workers, assuming the cache backend supports atomic add behavior.

Configure a Backend That Actually Supports This

This pattern only works if the backend is shared and atomic. A local-memory cache is not sufficient. A Redis-backed cache is a common choice:

1CACHES = {
2    "default": {
3        "BACKEND": "django.core.cache.backends.redis.RedisCache",
4        "LOCATION": "redis://127.0.0.1:6379/1",
5    }
6}

The shared backend is what makes the lock visible to every worker process.

Lock Expiry Is Essential

Always set a timeout on the lock. If a worker crashes after acquiring it, a lock with no expiry can block the task forever.

The timeout should be:

• long enough for normal completion
• short enough that a stale lock eventually clears

If the task duration varies a lot, you may need a more advanced heartbeat or renewable-lock strategy.

Argument-Specific Singleton Behavior

If you want only one task per key or per argument set, incorporate the arguments into the lock:

1@shared_task
2def sync_customer(customer_id):
3    lock_key = f"lock:sync_customer:{customer_id}"
4    acquired = cache.add(lock_key, "1", timeout=900)
5    if not acquired:
6        return "already-running"
7
8    try:
9        # do sync work
10        return "completed"
11    finally:
12        cache.delete(lock_key)

Now two different customers can sync concurrently, but the same customer cannot be processed twice at once.

Third-Party Helpers Can Reduce Boilerplate

Packages such as task-singleton helpers can wrap this pattern for you, but they are still built on the same underlying idea: a shared lock outside the worker process.

That can be worth it if your project uses many singleton tasks. If you only need one or two, an explicit lock often stays easier to reason about.

Prevent Queue Buildup If Necessary

There is a second problem beyond simultaneous execution: duplicate task scheduling. Even if the lock blocks duplicates from running, repeated enqueueing can still pile up many skipped tasks.

If that matters, consider deduplicating before enqueue, or use a task pattern that checks whether equivalent work is already scheduled or active.

Common Pitfalls

The biggest pitfall is using an in-memory Python variable as the lock. That only protects one process, not the Celery fleet.

Another common issue is forgetting lock expiry. If a worker dies, the job can become permanently blocked behind a stale lock.

People also use one global lock when they really need a per-argument lock, which accidentally serializes unrelated work.

Finally, skipping duplicate execution is not the same as preventing duplicate queue entries. Decide whether you need one or both protections.

Summary

• To run only one Celery task instance at a time, use a real distributed lock.
• Redis-backed cache with atomic add is a practical building block.
• Put the lock release in finally and always use an expiry timeout.
• Include task arguments in the lock key if singleton behavior should be per resource rather than global.
• Consider deduplication at enqueue time if duplicate queued tasks are also a problem.