Distributed Locks, and the Argument About Them

Topics Covered

What a Lock in Redis Actually Is

The one correct acquisition command

Why the token must be unique

Why release needs a script

What the TTL is for, and what it costs

The three properties a lock should have

Acquisition with a timeout

Extending a lock

Reentrancy, and whether you want it

What the lock does not know about

A complete minimal implementation

Where the lock key should live

What to take from the mechanism

A lock in Redis is a key whose existence means somebody holds it. Everything interesting about distributed locking follows from how thin that is, and from what a key with a TTL can and cannot promise.

The one correct acquisition command

SET resource token NX PX 30000. NX makes it create-only, so exactly one client can succeed. PX sets a millisecond expiry, so the lock disappears if the holder never releases it. token is a value unique to this acquisition.

Those three elements are all load-bearing and the command does all of it atomically, which is why this is the only form worth using. The older recipes that did SETNX followed by EXPIRE left a window in which a crash between the two commands produced a lock with no expiry, held forever, which is the worst failure the whole mechanism can have.

Why the token must be unique

Without a unique value, any client can release any lock. A client whose lock expired during its work, with another client having since acquired it, will delete the new holder's lock when it finishes and calls DEL.

That sequence is not hypothetical; it is the ordinary consequence of any pause longer than the TTL, and the pauses that cause it are commonplace. The token makes release conditional on still being the holder, which turns the bug into a detectable no-op. A random value of sufficient length, generated per acquisition, is all it takes.

Why release needs a script

Releasing correctly means checking that the value matches and deleting only then, which is a read-then-decide and therefore a script. Four lines: if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) else return 0 end.

A GET followed by a conditional DEL from the client is wrong for the familiar reason: between the two, the lock can expire and be acquired by somebody else, and the DEL then removes their lock. The script closes that window because nothing runs between its read and its write. This is the clearest small example of why the previous lesson matters.

All three elements are load-bearing and the single command does all of it atomically, which is why this is the only acquisition form worth using.

What the TTL is for, and what it costs

The TTL exists so a crashed holder does not block the resource forever. Without it, a client that dies mid-work leaves a lock nobody can release, and recovery requires manual intervention. With it, the lock releases itself.

The cost is precisely stated: the TTL is a promise that the holder will finish within it, and nothing enforces that promise. If the work takes longer, the lock expires while the holder still believes it holds it, and two clients are inside the critical section. The TTL converts an availability problem into a correctness problem, and that trade is the whole subject of this lesson.

The three properties a lock should have

Mutual exclusion, so at most one client holds it at a time. Deadlock freedom, so a lock is eventually released even if a holder crashes. And fault tolerance, so clients can acquire and release as long as enough of the lock service is up.

This is the framing the Redlock documentation uses and it is a useful one. The single-instance recipe above gives mutual exclusion in the absence of pauses, gives deadlock freedom through the TTL, and gives no fault tolerance at all, since the instance holding the lock is a single point of failure. Each of the next sections is about one of those gaps.

Acquisition with a timeout

A client that cannot acquire has three options: fail immediately, retry with a bound, or block. Retrying with a small randomised delay and a total deadline is the usual choice, and the randomisation matters for the same reason it does anywhere else.

Blocking indefinitely is almost always wrong, because it converts contention into an unbounded queue of waiting requests and the waiting consumes resources in the client. A deadline, after which the operation fails and reports that it could not get the lock, is both more honest and easier to operate, since the failure is visible rather than appearing as latency.

Extending a lock

A holder that expects to need more time can extend the TTL, again conditionally on still being the holder, with a script that compares the token and calls PEXPIRE. Some libraries do this on a timer in a background thread, calling it a watchdog.

Extension helps and does not solve anything. It reduces the probability that the TTL expires during legitimate work, which is useful, and it cannot help in the case that matters: if the client is paused, the watchdog is paused too, so exactly when the extension is needed it does not happen. That asymmetry is worth holding onto for the next section.

Reentrancy, and whether you want it

A lock that the same holder can acquire again, counting the acquisitions and releasing on the last, is reentrant. Implementing it in Redis means storing a count alongside the token, which a hash makes straightforward.

Whether to want it is a design question. Reentrancy exists to make nested calls within one process work, which in a distributed setting usually indicates that the lock's scope is unclear. The simpler arrangement is a lock acquired at one well-defined boundary and held across everything inside it, which needs no counting and is much easier to reason about.

What the lock does not know about

The resource being protected has no idea the lock exists. Redis holds a key; the database row, the file, the external API being guarded knows nothing, enforces nothing and will happily accept a write from a client whose lock expired.

This is the structural fact that the rest of the lesson keeps returning to. A lock is advisory by construction: it coordinates willing participants and cannot stop an unwilling or confused one. Any argument about distributed locking that does not account for this is arguing about the wrong thing, which is why fencing, introduced shortly, is the concept that actually matters.

A GET followed by a conditional DEL from the client is wrong for the familiar reason, and the script closes the window because nothing runs between its read and its write.

A complete minimal implementation

Acquire with SET resource token NX PX ttl. Do the work. Release with the compare-and-delete script. On failure to acquire, retry with jitter until a deadline, then fail. Optionally extend with a compare-and-expire script if the work may run long.

That is the whole of a correct single-instance lock and it is about thirty lines in any language. Most of the complexity in lock libraries beyond this is either Redlock, which the third section covers, or convenience around the retry policy. If a library does something materially more complicated than this for a single instance, it is worth understanding why before trusting it.

Where the lock key should live

One key per protected resource, named after the resource rather than after the job doing the protecting, so that two different code paths guarding the same thing cannot accidentally choose different keys.

That sounds obvious and it is the most common real defect after the missing token. A lock named after the worker, or carrying a job identifier in it, protects nothing, because the second worker computes a different key and acquires it happily. Naming the key from the resource identity, with a short fixed prefix so locks are findable in the keyspace, removes the whole class of mistake.

What to take from the mechanism

Three things. SET with NX and PX and a unique token is the only correct acquisition, and release must be a compare-and-delete script. The TTL buys deadlock freedom and pays for it by making expiry-during-work possible, which is a correctness problem rather than an availability one. And the lock is advisory: the protected resource does not know it exists and will not enforce it.

Key Insight

The TTL is the trade at the centre of all distributed locking. Without it, a crashed holder blocks the resource until somebody intervenes. With it, the lock can expire while the holder is still working, putting two clients inside the critical section. So the TTL converts an availability problem into a correctness problem, and every argument in this lesson is about what to do with that.