Paxos leader election might not terminate
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Paxos is a protocol for solving consensus in a network of unreliable processors. Consensus means that the group must agree on a single result even in the face of faults. The protocol was proposed by Leslie Lamport and is widely recognized for its fault-tolerant capabilities which are crucial in distributed systems. One component of the Paxos algorithm involves leader election, which can sometimes pose issues where the protocol might not terminate efficiently or at all. Understanding why this non-termination can occur is paramount for systems depending heavily on consensus protocols.
Understanding Paxos Leader Election
In Paxos, a leader is elected to propose values which the other nodes in the network can agree on. The leader election is crucial because having a stable leader can reduce the message complexity and increase the efficiency of reaching consensus. The basic steps in the Paxos leader election are:
- A node (proposer) asserts itself as a leader by sending a "prepare" message, which includes a proposal number to all other nodes (acceptors).
- Acceptors respond to the "prepare" notice if the proposal number is higher than any they've seen before. They promise not to accept any earlier proposals.
- If the proposer receives enough "promises" from a majority of acceptors, it sends an "accept" request with a proposal value.
Factors Leading to Non-Termination
The leader election process in Paxos can fail to terminate under certain conditions, primarily due to failures and the asynchronous nature of the system. The major factors include:
Message Loss or Delay
In distributed systems, messages can be lost or significantly delayed. If a critical number of "prepare" or "promise" messages are lost or delayed, it prevents a proposer from moving to the next step or becoming a leader, causing the election process to stall or retry indefinitely.
Simultaneous Leadership Assertions
Concurrency can lead to multiple nodes trying to become leaders approximately at the same moment, each issuing its own proposal number. If these numbers continually overtake each other, it could lead to a situation where no single proposal is supported by a majority, known as a "contention storm".
Rapid Changes in Network Membership
In highly dynamic networks where nodes frequently join or leave, maintaining a consistent set of acceptors can be challenging. Changes in the group of acceptors in the midst of an election can invalidate the process, requiring a new round of elections.
Failure and Recovery of Nodes
Failure of a node, especially if it's a proposer during the election, can interrupt the process. If a node fails after sending a "prepare" message but before receiving all "promises", or if acceptors who have responded with a promise fail, it can halt progression.
Example Scenario
Consider a system with three nodes A, B, and C. If A and B simultaneously propose to become leaders, while C, being slow, alternates between responding to A and B:
- A sends a prepare with proposal number 1.
- B sends a prepare with proposal number 2 immediately after.
- C receives A's proposal and promises A but then receives B's, which is higher, so promises B.
- A increases its proposal number and retries, overtaking B, to which C again shifts its promise.
This cycle can potentially continue indefinitely, especially if timings overlap closely.
Summary Table
| Factor | Description Impact on Termination |
| Message Loss or Delay | May prevent a proposer from gathering enough promises, thus stalling the process. |
| Simultaneous Leadership Assertions | Can cause proposals to continually overtake each other, preventing a clear majority. |
| Rapid Changes in Network Membership | Changes in acceptor set can invalidate ongoing election processes, necessitating restarts. |
| Failure and Recovery of Nodes | Failures can interrupt the election process or prevent the collection of sufficient promises. |
Conclusion
The leader election mechanism in Paxos is integral to its ability to achieve consensus, but due to its sensitivity to timing and failures, it can occasionally fall into non-termination scenarios. Understanding these scenarios is essential when designing systems that rely on Paxos, ensuring either mitigation strategies are in place or choosing alternative protocols where appropriate.
Related reading
- Paxos questions if proposer down, what happened?
- Paxos understanding
- paxos vs raft for leader election
- paxos why do ids have to increase monotonically?
- PBFT consensus algorithm and double spending
- Peak-finding algorithm for Python/SciPy
- PBFT view-change What happens to committed operations after the valid snapshot?
- PBFT Why cant the replicas perform the request after 2/3 have prepared? why do we need commit phase?

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.