Two Phase Commit blocking on coordinator failure
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
The Two Phase Commit (2PC) protocol is commonly employed in distributed systems to achieve consensus among all participating nodes (often databases or transaction managers) on whether to commit or abort a transaction. By design, the protocol is intended to ensure that, despite failures, either all nodes commit the transaction (thus maintaining consistency) or all abort, thereby preserving atomicity across the system. However, 2PC is vulnerable to blocking in the event of certain failures, particularly failures affecting the coordinator.
Understanding Two Phase Commit
Let’s break down the two phases of the 2PC protocol:
- Prepare Phase:
- The coordinator asks all participating nodes (cohorts) to prepare (lock) the transaction resources, ensure that the transaction can be committed, and vote on commitment.
- Each cohort can vote 'yes' (If it can commit without issues) or 'no' (if issues are found, like violation of constraints or any kind of failure during preparation).
- Commit Phase:
- If all cohorts vote 'yes', the coordinator sends a 'commit' command.
- If any cohort votes 'no', the coordinator sends an 'abort' command to all cohorts.
Impact of Coordinator Failure
The blocking issue arises if the coordinator fails permanently or for an extended period during the critical juncture between the end of the first phase and the completion of the second phase.
Scenario Analysis:
- After Prepare before Commit/Abort: Suppose the coordinator crashes after receiving all 'yes' votes but before instructing nodes to commit. In this case, cohorts are left in a locked state, waiting for a command to either commit or rollback the transaction. This state of limbo impedes accessibility and can lead to a complete system halt if the coordinator does not recover.
- Midway Through Sending Commit/Abort Commands: If the coordinator fails while sending commit commands, some nodes might have committed, while others have not yet received or acted on the command. This partial commitment violates the atomicity property of transactions.
To detail the importance of coordinator and participant states, here’s a summarizing table:
| Phase | Coordinator Action | Cohort State Before Action | Cohort State After Action | System State on Coordinator Failure |
| Prepare | Send prepare to all | Active | Prepared (locked) | Blocking: Cohorts await command |
| Commit | Send commit to all | Prepared | Committed or committing | Partially Committed: Non-atomic transactions |
| Abort | Send abort to all | Prepared | Aborted | Safe but resources locked until recovery |
Resolving Coordinator Failures
Several strategies can be applied to handle or mitigate the impact of coordinator failure:
Timeout Mechanism
Cohorts can implement a timeout mechanism. If the expected command (commit/abort) from the coordinator does not arrive within this timeout period, cohorts can autonomously decide to abort the transaction. However, this approach doesn't safeguard against partially committed states where some cohorts might have executed the commit command.
Logging and Recovery
The coordinator can log the state of each transaction persistently. In case of a failure and subsequent recovery, the coordinator can review these logs to decide the next steps to ensure all nodes reach a consistent state.
Use of a Backup Coordinator
Designing the system with a backup coordinator that can take over if the primary fails between phases can help. The backup would need to maintain the same information as the primary to seamlessly continue from the point of failure.
Distributed Consensus Protocols
Moving away from the 2PC to more robust consensus protocols like Paxos or Raft might be appropriate. These protocols are designed to handle failures more gracefully and continue operation without blocking, as they do not require a single point of failure like a central coordinator.
In conclusion, while the Two Phase Commit protocol is elementary and useful in maintaining atomicity across distributed systems, it's crucial to be aware of its limitations, particularly the susceptibility to blocking upon coordinator failure. Implementing robust fault-tolerant mechanisms and potentially considering alternative consensus protocols are central to designing resilient distributed systems.
Related reading
- Two phase commit what happens if the coordinator dies between sending two confirmations
- Two single-column indexes vs one two-column index in MySQL?
- TypeError db.collection is not a function
- TypeError only integer scalar arrays can be converted to a scalar index with 1D numpy indices array
- Unable to merge dex
- Unable to resolve unable to get local issuer certificate using git on Windows with self-signed certificate
- Types in MySQL BigInt20 vs Int20
- Unable to acquire JDBC Connection

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.