kafka Multi-Datacenter with high availability
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Apache Kafka is a distributed streaming platform known for its high throughput, built-in partitioning, replication, and inherent fault tolerance. This makes it an excellent choice for managing high availability (HA) across data centers. A multi-data-center deployment model is often required for organizations looking to increase the system's resilience and availability on a global scale while keeping latency low and ensuring data locality.
Architecture Overview
Kafka employs a publisher-subscriber model where messages are persisted in topics that are split across different partitions. These partitions are replicated across multiple nodes (brokers) to ensure data redundancy. A Kafka cluster can span multiple data centers. Here, the architecture is designed to handle failures of entire data centers and still be able to serve data to consumers without loss.
Key Configurations for Multi-Datacenter Setup
1. Replication Factor: Set the replication factor in Kafka to ensure that each message is stored on multiple brokers across different geographic locations. This helps in preventing data loss.
2. Min.insync.replicas: This parameter dictates the minimum number of replicas that must acknowledge a write for it to be considered successful. For high availability, this number should be set to at least 2.
3. Rack Awareness: Configuring Kafka brokers with the knowledge of the physical (or virtual) rack where they are located in can manage the replica distribution such that replicas for partitions are spread across different data centers.
Implementation Details
Synchronizing Data Across Data Centers
Mirroring: Kafka's MirrorMaker tool is used to replicate a source Kafka cluster into a target Kafka cluster, which could be located in a different data center. It subscribes to topics in the source cluster and publishes to the same topics in the destination cluster.
Example usage:
Cross-Cluster Failover:
For failover and disaster recovery, you may set up a robust mechanism where each data center acts as both primary and disaster recovery (DR) for various services. In case one of the data centers fails, the other can take over seamlessly.
Considerations for Latency and Fault Tolerance
Latency is a crucial factor in the performance of distributed systems like Kafka. Having multiple data centers can increase the inter-data center latency. Properly mapping and understanding consumer locations relative to data center locations should guide the data replication and partitioning strategy to minimize latency.
Additionally, effectively handling fault tolerance involves ensuring that partitions and their replicas are not just distributed across different servers but across geographical boundaries.
Summary Table
| Key Element | Description |
| Replication Factor | Determines number of copies of data. Increase to enhance data durability across data centers. |
| Min.insync.replicas | Minimum number of replicas that must acknowledge a write; crucial for consistency. |
| Rack Awareness | Helps in distributing replicas of a Kafka topic's partitions across physical locations. |
| Latency Concerns | Prioritize geographic placement of data centers to manage latency. |
| Fault Tolerance | Spread data logically and geographically to avoid data loss in the event of a site failure. |
Use Cases
Multinational corporations often require real-time data feeds accessible across continents with strict regulatory and compliance needs secluding data geographically. Implementing multi-datacenter high availability can meet these demands by ensuring continuous operation and data availability even during partial system failures.
Monitoring & Managing Health
Monitoring tools like Prometheus, combined with Grafana dashboards, offer insights into Kafka metrics such as throughput, latency, and error rates across data centers. These are invaluable in managing and healing any issues preemptively before they impact service availability.
In conclusion, creating a Kafka deployment across multiple data centers involves considerations around replication, data synchronization, fault tolerance, and latency management. These efforts ensure a deployment is resilient, meets performance demands, and complies with data governance standards. When well-implemented, Kafka's robust framework can handle failures gracefully and maintain high standards of data availability and integrity.
Related reading
- Kafka Multi Node setup Unreasonable length in Zookeeper logs
- Kafka multiple consumers for a partition
- Kafka multiple partition ordering
- Kafka multiple topic consume
- kafka new producer is not able to update metadata after one of the broker is down
- Kafka Number of Partitions are more than no of broker
- Kafka new producer timeout
- Kafka No broker in ISR for partition

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.