Design A Distributed Locking System - System Design

System requirements

Functional:

Acquire Lock: A client can acquire a lock on a resource.
Release Lock: A client can release a lock when done.
Timeout / TTL: Locks automatically expire to prevent deadlocks.
Fairness: Requests should be served roughly in order (FIFO) when possible.
Concurrent Clients: Multiple clients can request locks concurrently.

Non-Functional:

High Availability: System continues working despite node failures.
Scalability: Can handle increasing numbers of locks and clients.
Low Latency: Fast lock acquisition and release.
Consistency: A resource is locked by at most one client at a time.

Capacity estimation

70k ops/sec workload (example 100k clients, 5 locks, TTL=10s).
Each lock record ~120B; 1M locks = ~120MB RAM.
Network: ~63 MB/sec for replication.
4–6 Raft clusters (3–5 nodes each) for horizontal scaling.
SLA: <10ms acquire/release, <50ms P99 latency.

API design

POST /lock/acquire

{ "resource": "X", "client_id": "C1", "ttl": 5000 }

POST /lock/release

{ "resource": "X", "lease_id": "abc123" }

POST /lock/renew

{ "lease_id": "abc123", "ttl": 5000 }

resource: The unique identifier of the thing you want to lock (e.g., "file_123", "user:42").

client_id: The ID of the client (or session) requesting the lock.

ttl: Time-to-live (in milliseconds). How long the lock should be valid before it auto-expires.

Database design

KV stores like etcd, Consul, ZooKeeper, Redis provide:

CAS (Compare-And-Swap) or SETNX (set if not exists).
Leases with TTL (auto-expire).
Linearizable reads/writes (strong consistency).
Fast performance (in-memory with persistence).

/locks/
  resource_X = {"owner_id":"C1","lease_id":"abc123","expires_at":1700000123}
/queues/
resource_X/
1695055500000_c2 = {"client_id":"C2","requested_ttl":5000}
1695055501000_c3 = {"client_id":"C3","requested_ttl":5000}
/leases/
abc123 = {"resource_key":"resource_X","owner_id":"C1","expires_at":1700000123}

High-level design

Consensus Metadata Store (Raft cluster like etcd): authoritative lock state & durable FIFO queues, fencing token generation, wait-for graph for detection. Guarantees linearizability.

API Gateway (stateless): accept client requests, forward to metadata store (or local cache for granted locks), handle renewals, timeouts, and metrics.

Clients : robust client library with heartbeat/renew, local retry/backoff, and API wrappers.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?