Introduction

Celery Beat can schedule a task every minute, but it does not guarantee that the previous run has already finished. If one run overlaps the next, you can end up with duplicate work, conflicting writes, or resource contention, so the usual solution is to add an external lock around the task body.

Why Celery Beat Does Not Solve This for You

Celery Beat is only a scheduler. It enqueues tasks according to a timetable. Workers then execute those tasks independently, and if a task runs longer than its schedule interval, a second copy may start before the first one ends.

So the problem is not really "make Beat smarter." The problem is "make task execution mutually exclusive."

Use a Distributed Lock

In a multi-worker deployment, the safest pattern is a lock in a shared system such as Redis. That lets every worker check the same lock state before doing the task.

1import time
2
3import redis
4from celery import Celery
5
6app = Celery("tasks", broker="redis://localhost:6379/0")
7redis_client = redis.Redis(host="localhost", port=6379, db=1)
8
9LOCK_KEY = "lock:nightly-report"
10LOCK_TTL_SECONDS = 300
11
12
13@app.task(bind=True)
14def nightly_report(self):
15    lock = redis_client.set(LOCK_KEY, self.request.id, nx=True, ex=LOCK_TTL_SECONDS)
16    if not lock:
17        print("Task already running, skipping this schedule tick")
18        return
19
20    try:
21        print("Generating report...")
22        time.sleep(10)
23        print("Report finished")
24    finally:
25        owner = redis_client.get(LOCK_KEY)
26        if owner and owner.decode() == self.request.id:
27            redis_client.delete(LOCK_KEY)

This pattern uses:

• 'nx=True so the key is created only if it does not already exist'
• 'ex=... so the lock expires automatically if the worker crashes'
• ownership check before delete so one task instance does not accidentally remove another instance's lock

Why a Plain In-Memory Flag Is Not Enough

A global Python flag only works inside one worker process. As soon as you run multiple workers, multiple containers, or separate machines, that local state stops protecting anything.

That is why Celery single-instance control almost always means:

• Redis lock
• database lock
• or a purpose-built library that uses one of those mechanisms underneath

Libraries and Alternatives

You do not have to build this from scratch every time. Libraries such as celery-singleton or custom task base classes can wrap the same locking pattern for you. The important design choice is still the same: use a lock visible to all workers.

If you need stronger guarantees, a database row lock or advisory lock can also work, but Redis is often simpler operationally for scheduled-task exclusion.

Timeouts Matter

A lock without expiration can deadlock the schedule forever if a worker dies mid-task. A lock with an expiration that is too short can allow overlap because the next scheduled run acquires the key before the first execution is actually done.

So choose a TTL that is:

• longer than normal execution time
• but not so long that stale locks block the system for hours

For long-running tasks, refreshing the lock periodically can be better than using one large static TTL.

Common Pitfalls

• Assuming Celery Beat itself prevents overlap.
• Using a local in-process flag instead of a distributed lock.
• Forgetting a lock expiration and leaving the system vulnerable to stale locks after crashes.
• Deleting the lock unconditionally without verifying ownership.
• Choosing a TTL shorter than the real runtime of the task.

Summary

• Celery Beat schedules tasks, but it does not prevent concurrent task instances.
• Use a distributed lock, typically in Redis, to guarantee one active execution at a time.
• Add a TTL so worker crashes do not leave stale locks forever.
• Verify lock ownership before deleting the key.
• The real solution is task-level mutual exclusion, not scheduler-level wishful thinking.