Introduction

In the world of distributed databases, maintaining high availability and data integrity across multiple nodes is paramount. In setups involving master-slave architectures, a common operation is to promote a secondary (or slave) node to a primary (or master) node. This is particularly relevant in scenarios where the primary node fails or requires maintenance, and the system needs to continue functioning seamlessly. This article explores the technical procedures and considerations involved in making a secondary node primary, including both manual and automated approaches.

Understanding Node Roles

Primary Node

• Purpose: Handles all write operations and coordinates data across secondary nodes.
• Responsibilities: Maintains the latest and most complete view of the data.

Secondary Node

• Purpose: Receives read operations and asynchronously fetches updates from the primary node.
• Responsibilities: Acts as a backup for failover scenarios and enhances read scalability.

Scenarios for Promotion

1. Primary Node Failure: The most common trigger for promoting a secondary node is the failure of the primary node. A quick promotion ensures minimal downtime and data loss.
2. Maintenance Tasks: Sometimes a primary node needs to be taken down for maintenance or upgrades. Promoting a secondary node can keep the system operational during this period.
3. Geographic Considerations: In geodistributed setups, promoting a secondary node closer to a particular user base can reduce latency for local users.

Technical Steps for Promoting a Secondary Node

Manual Promotion

In a manual promotion setup, administrators are responsible for the switch:

1. Identify Potential Candidates: Choose a candidate secondary node. It should be the most up-to-date in terms of replication lag.
2. Assess Replication Lag: Check the replication status to ensure the secondary node has the latest data.
3. Stop Applications: Temporarily halt applications or set them to read-only mode to prevent discrepancies during promotion.
4. Promote the Secondary Node: Configurations vary depending on the database system (e.g., PostgreSQL or MySQL), but typically involve commands or config changes like:
   • For MySQL: Use RESET MASTER and SET GLOBAL read_only=0 to allow write operations.
   • For PostgreSQL: Modify recovery.conf to remove recovery settings.
5. Reconfigure Old Primary and Other Secondaries: Once promoted, adjust configurations on the old primary and other nodes to realign them with the new primary.

Automated Promotion

Many modern databases have built-in support for automated failover processes that handle node promotion:

1. Heartbeat Monitoring: Utilizes a watcher service to monitor the health of the primary node.
2. Automatic Election: Upon primary failure detection, a process often aided by a consensus protocol (like Raft or Paxos) will elect a new primary.
3. Notification Services: Alerts and logging to notify administrators of the changeover, maintaining transparency and records of the event.

Practical Example

Consider a PostgreSQL setup with synchronous replication between nodes. Here's a simplified manual process to promote a secondary node:

1# On the secondary node to be promoted
2sudo systemctl stop postgresql
3cp /etc/postgresql/12/main/recovery.conf /etc/postgresql/12/main/recovery.done
4
5# Restart to activate primary mode
6sudo systemctl start postgresql

After this, reconfigure other secondary nodes to follow the newly promoted primary node.

Considerations and Challenges

• Replication Consistency: Ensure all transactions are committed across nodes.
• Network Partitioning: Protect against split-brain scenarios by confirming network stability.
• Data Integrity: Utilize tools to verify data consistency post-promotion.

Summary Table

Conclusion

Promoting a secondary node to primary is an essential procedure for maintaining database availability and reliability. Whether manually or automatically triggered, understanding the nuances of this process helps administrators ensure smooth transitions with minimal impact on users. Careful monitoring and testing are fundamental to mastering this aspect of database management.