distributed storage why the redundant copy is 3 by default instead of 2?
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
In the realm of distributed storage systems, data reliability, availability, and fault tolerance are paramount. One common approach to achieving these goals is through data replication. The default choice for the number of replicas in many distributed systems, such as Hadoop's Distributed File System (HDFS), is often three. This choice is not arbitrary and has strong technical and practical justifications.
Why Three Replicas?
The core reason for storing three copies of each data block instead of two is to strike a balance between data availability, fault tolerance, and cost in scenarios of node failure or corruption.
1. Fault Tolerance
Replicating data three times reduces the risk of data loss. With single replication, any disk failure means immediate data loss. With two replicas, if one replica is lost due to a failure, the system is left with only one copy, entering a vulnerable state where any additional failure can lead to data loss. Three replicas ensure that even if one replica fails, there is a higher probability of maintaining system functionality and data integrity until the lost or corrupted replica is replaced.
2. Simultaneous Node Failures
Distributed systems are designed to handle the fault tolerance not just against isolated incidents but also against simultaneous failures. Two replicas do not safeguard against a scenario where both nodes fail almost simultaneously or within the recovery window of each other. Three replicas increase the resilience to simultaneous failures.
3. Data Recovery and Rebalancing
With three replicas, the system has more flexibility and options for data recovery and rebalancing. If one node fails, the system can choose between the two remaining replicas to regenerate the missing data, optimizing recovery time and load across the network.
4. Improving Read Availability
In many distributed systems, data is often read more frequently than it is written. Storing data on three different nodes allows the system to load balance read requests more effectively, potentially reducing access times and improving overall system performance.
A Simple Experiment:
Consider a cluster that stores data with replication factor of 2 and 3:
- Scenario with Replication Factor 2:
- If one node out of 100 fails, there is a significant risk that some data will become unavailable due to the potential loss of the second copy before rebalancing.
- Scenario with Replication Factor 3:
- The probability of losing all copies of data becomes significantly lower even if two nodes fail at the same time. The system still has one more replica to maintain data integrity.
Key Point Summary
Here's a table summarizing why three replicas are generally preferred over two in distributed storage systems:
| Factor | Replication Factor 2 | Replication Factor 3 |
| Fault Tolerance | Low (Two simultaneous node failures can result in data loss) | High (Can handle up to two simultaneous node failures) |
| Availability | Medium (Higher risk during node failures) | High (Higher availability during rebalancing) |
| Cost | Lower (Less storage space required) | Higher (More storage space required) |
| Performance | Lower (Potential bottlenecks during recovery) | Higher (More options for load balancing reads) |
Conclusion
In conclusion, while setting the replication factor to three by default increases storage requirements, it significantly enhances data reliability, availability, system recovery efficiency, and fault tolerance in distributed storage systems. This setup ensures that even in the face of simultaneous node failures or in high-load scenarios, the probability of data loss or severe performance degeneration is minimized. Thus, despite the higher resource cost, the benefits of robustness and system performance often outweigh the disadvantages, making three replicas a rational default setting in distributed storage systems.
