What is a good solution for cross datacenter master-master replication?
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Master-master replication across data centers is a sophisticated strategy designed to enhance the resilience, availability, and performance of database systems spread across geographical locations. This replication strategy allows data to be updated simultaneously in two or more locations, supporting high availability and disaster recovery. Here, we delve into the mechanisms, technical considerations, and potential solutions for implementing effective cross-datacenter master-master replication.
Understanding Master-Master Replication
Master-master replication, also known as bidirectional replication, is a method where two or more database servers function as masters, allowing read and write operations on each. Changes made on one master are automatically replicated to the other, ensuring data consistency and coherence across locations.
Key Characteristics:
- Bidirectional Data Flow: Each server in master-master replication can initiate data changes, which are then replicated to all other master nodes.
- Conflict Resolution: Since both nodes accept write operations, conflict resolution mechanisms are essential to handle divergent changes to the same data point.
- High Availability: The system can continue to operate even if one master fails, as other masters can handle requests.
Technical Considerations
Network Latency and Throughput
Cross datacenter replication involves data transmission over wide area networks (WAN), which can have higher latency and lower throughput compared to local networks. Efficient replication needs a robust network setup with optimized latency and high throughput capabilities to handle synchronous data replication efficiently.
Conflict Resolution Mechanisms
In a master-master setup, conflicts can arise when the same dataset is modified simultaneously in different locations. Implementing a reliable conflict resolution strategy is crucial. Common methods involve using timestamps, version numbers, or last write wins logic to resolve conflicts.
Scalability and Performance
Replicating data across data centers can impact database performance due to the overhead of synchronization and conflict resolution. It’s important to assess performance impacts and scale resources accordingly.
Technologies and Solutions for Master-Master Replication
Several technologies offer support for cross datacenter master-master replication, each with its own advantages and limitations:
- MySQL Cluster CGE: Provides synchronous multi-master replication with automatic conflict detection and resolution, suitable for applications requiring immediate consistency.
- Galera Cluster for MySQL: Offers synchronous replication through a certification-based approach, reducing the chance of conflicts with high system availability.
- Oracle GoldenGate: A software package for real-time data integration and replication in heterogeneous database environments, supporting conflict detection and resolution policies.
- CouchDB: Utilizes Multi-Master replication at the database level suitable for eventual consistency models, easy to scale, and can handle diverse data types.
- MongoDB: Supports built-in replication and high availability with automated conflict resolution, although traditionally set in a primary-secondary node configuration, it can be configured for multi-master replication in special scenarios.
Here is a table summarizing some potential solutions:
| Technology | Type | Conflict Resolution | Pros | Cons |
| MySQL Cluster CGE | Synchronous | Automatic conflict resolution | Real-time updates, data consistency | Requires a lot of resources |
| Galera Cluster | Synchronous | Certification-based conflict handling | High availability, no data loss | Performance penalty under heavy loads |
| Oracle GoldenGate | Asynchronous | Custom policies | Flexibility, supports heterogeneous systems | Complex setup |
| CouchDB | Multi-Master | Revision-based | Good for distributed web apps | Eventual consistency only |
| MongoDB | Primary-Secondary | Update conflicts resolved by timing | High performance, wide feature set | Not native master-master |
Best Practices
- Choosing the Right Tool: Depending on the specific use case, whether high availability, consistency, or minimal latency is the priority, choose the technology that best aligns with your requirements.
- Regular Monitoring and Testing: Continuous monitoring of replication processes and periodic tests to ensure data consistency and integrity across data centers.
- Optimize Network Configuration: Use dedicated and optimized network links for replication traffic, and configure proper QoS policies to prioritize this critical data traffic.
Conclusion
Cross datacenter master-master replication is a powerful strategy for organizations that require high availability and robust disaster recovery mechanisms. By understanding the technical requirements and choosing the appropriate solution, businesses can achieve an effective replication setup that meets their operational, data consistency, and performance needs.
Related reading
- What is a headless service, what does it do/accomplish, and what are some legitimate use cases for it?
- What is a partition leader in Apache Kafka?
- What is a process for recovering a failed master from a slave with PostgreSQL?
- What is a TTL 0 in CloudFront useful for?
- What is a Kubernetes Manifest?
- What is a provisioning profile used for when developing iPhone applications?
- What is Bulkhead Pattern used by Hystrix?
- What is CRDT in Distributed Systems?

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.