Design a Distributed Locking System

You build a distributed system where two services each run a cron job that processes payments. Both fire at midnight. Without coordination, both process the same batch: customers get charged twice. A distributed lock solves this: one service acquires the lock, processes payments, and releases it. The other waits or skips. The challenge is not building the lock. The real challenge is making the lock survive server crashes, network partitions, and clock drift.

Functional

Acquire and release locks on named resources: The core guarantee is mutual exclusion. At most one client holds a given lock at any time. Acquiring returns a unique token that the client uses to release or renew. Releasing frees the resource for the next waiter.

Lock state is tiny, and that is the key architectural enabler. Each lock record is roughly 200 bytes: resource name, owner ID, lock token, TTL, and timestamps. With 1 million concurrent locks, total state is 200 MB: fits trivially in Redis memory. The throughput requirement drives the technology choice more than the storage requirement.

Throughput

10,000 lock operations per second is the baseline. Redis handles 100,000 SET operations per second on a single node, a single Redis instance has 10x headroom for lock operations. ZooKeeper handles roughly 10,000 writes per second: functional but with no margin for spikes. PostgreSQL with SELECT FOR UPDATE handles roughly 5,000 lock operations per second due to row locking overhead.

Three operations define the entire interface: acquire, release, renew. The details matter more than the names. The acquire call must include a TTL so crashed holders do not block forever, an owner ID so only the owner can release, and optionally a timeout so waiters do not hang indefinitely.

Core APIs

Acquire lock: POST /locks/{resource_name}/acquire

Two storage layers serve different purposes. Redis is the fast path: it holds active lock state in memory for sub-millisecond access. PostgreSQL is the durable path: it stores lock metadata, audit history, and serves as the recovery source when Redis restarts.

Redis Data Model

Active lock key: lock:{resource_name}

Four components, one critical invariant: the Lock Service is the only component that grants locks. It always does so via atomic check-and-set. Everything else, the cache, the queue, the audit log: supports this core operation.

API Gateway: Handles rate limiting, authentication, and request routing. Protects the Lock Service from request floods during mass service restarts. Rate limits are per-client to prevent a single misbehaving service from overwhelming the lock system.

Two flows define this system. The acquire-release flow is the happy path, a client gets the lock, does work, and releases. The renewal flow is the safety path, a background watchdog extends the TTL so a slow operation does not lose its lock mid-execution.

Acquire Flow

"What happens when two clients try to acquire the same lock at the exact same instant?" This is the core interview question. Client A and Client B both send SET resource owner NX to Redis. Only one can succeed. Redis processes commands sequentially on a single thread. But what happens if the Redis master crashes right after granting the lock to Client A, before replicating to the slave? The slave promotes, Client B tries again, and now two clients hold the same lock.

Atomic Check-and-Set (the GATE)

Three implementation approaches, each with a fundamental trade-off between speed and safety. Redis is the fastest but weakest during failover. ZooKeeper and etcd are the safest but slowest. PostgreSQL is the simplest but limited in throughput. Most production systems use Redis with fencing tokens: accepting the failover gap in exchange for sub-millisecond latency.

Redis (SET NX / Redlock)

Every failure maps to a specific risk: double-granting (two holders), indefinite blocking (no holder), or stale execution (expired holder still writing). The system is designed so that failures cause at most brief double-granting caught by fencing tokens or brief blocking resolved by TTL expiration. never silent data corruption.

Redis Master Failover

The master grants a lock to Client A, then crashes before replicating to the slave. The promoted slave has no record of Client A's lock. Client B acquires the same lock on the new master. Two clients now believe they hold the lock. Mitigation: Redlock across independent nodes eliminates the replication dependency. Fencing tokens at the protected resource reject Client A's stale writes even if Redlock is not used.

The system handles single-resource mutual exclusion at scale. The next frontier is richer lock types, smarter deadlock handling, and cross-datacenter coordination.

Read-Write Locks

The current system grants exclusive access for every operation. Many workloads have far more reads than writes. Read-write locks allow multiple concurrent readers but exclusive writers. For a configuration store where reads outnumber writes 100 to 1, this dramatically reduces contention. Implementation: use a Redis hash with a reader count field and a writer field. Read locks increment the count. Write locks require the count to be zero.