Writes on Cassandra Network Partitioned Nodes
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Apache Cassandra is a highly scalable, distributed NoSQL database designed to handle large amounts of data across many commodity servers. A key feature of Cassandra is its exceptional handling of network partitions. This article delves into how writes are managed in Cassandra when nodes are partitioned in the network.
Understanding Network Partitions in Cassandra
Network partitions occur when there is a failure in the network that prevents a group of nodes within a Cassandra cluster from communicating with other such groups. This could be due to network issues, hardware failures, or any disruptions in the communication path.
Cassandra is designed to be fault-tolerant and can continue to operate when a network partition occurs. It uses a peer-to-peer distributed system architecture, where each node communicates with each other without a central coordinator. This design helps in maintaining the database's availability and partition tolerance as per the CAP theorem (Consistency, Availability, and Partition Tolerance).
Write Operations During Network Partitions
When a client issues a write request to a Cassandra node and the cluster is experiencing a network partition, several scenarios can unfold based on the configured consistency level and the state of the partition:
Consistency Levels
Cassandra offers various consistency levels for operations. The choice of consistency level directly affects how writes are handled during partitions:
- ANY: The write operation must be accepted by at least one node in the cluster.
- ONE or TWO: The write must be acknowledged by one or two nodes, respectively, irrespective of their position concerning the partition.
- QUORUM: Requires a majority of the replica nodes for the data being written to agree on the write.
- ALL: All replicas for the data block being written must acknowledge the write for it to be considered successful.
Example of a Write Operation During a Partition
Consider a cluster with 5 nodes (N1, N2, N3, N4, N5) and a replication factor of 3. Let's say there’s a network partition between (N1, N2, N3) and (N4, N5).
- A write request is issued to N1 with a QUORUM consistency level.
- N1 tries to forward the write request to all replicas of the data. Due to the partition, it can only reach N2 and N3.
- Since QUORUM for a replication factor of 3 is 2 nodes (i.e., majority), the write is successful if at least two nodes (including N1 itself) acknowledge the write back to the client.
Handling Failure
Cassandra uses hinted handoff and read repair strategies to handle write failures due to network partitions:
- Hinted Handoff: When a node accepts a write but cannot forward it to other necessary replicas (due to a partition or node down), it stores a hint. Once the partition heals or the node is back up, the hint is used to complete the write.
- Read Repair: During subsequent reads of the same data, if discrepancies between replicas are found, Cassandra will initiate a read repair process to make all replicas consistent with the latest successful write.
Performance Considerations
Writing to a partitioned node cluster in Cassandra is heavily dependent on the chosen consistency level. Higher consistency levels (e.g., QUORUM, ALL) increase the latency of write operations but ensure more strong consistency of data across the nodes.
Below is a summary table that illustrates the dependency of write success on consistency levels during a network partition.
| Consistency Level | Required Acknowledgements | Success in Example Partition |
| ANY | 1 from any node | Yes |
| ONE | 1 from one node | Yes |
| TWO | 2 from two nodes | Yes |
| QUORUM | Majority (2 out of 3) | Yes |
| ALL | All replicas (3) | No |
Conclusion
Handling writes in the presence of network partitions is a complex aspect of distributed systems design. Cassandra offers configurable consistency levels to balance between availability, consistency, and the performance of write operations during such events. By understanding and appropriately configuring these levels, administrators can tune the system to meet specific use case requirements while navigating the challenges posed by network partitions.
Related reading
- Writes to geographically distributed database
- Wrong count with cassandra-cql
- Xcode 4 and Core Data How to enable SQL Debugging
- YCSB for Cassandra 3.0 Benchmarking
- You are trying to add a non-nullable field 'new_field' to userprofile without a default
- You can't specify target table for update in FROM clause
- You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application
- ZeroMQ vs Oracle queuing

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.