Historically, ZooKeeper has been a critical component of Kafka, responsible for managing the distributed system’s coordination tasks, such as leader election, metadata storage, and configuration management. However, with newer Kafka versions, particularly from Kafka 2.8 onwards, ZooKeeper is no longer required due to the introduction of the KRaft (Kafka Raft) consensus protocol.

Here’s a breakdown of when and why ZooKeeper is or isn’t needed:

When ZooKeeper Is Required

• Kafka Versions Below 2.8:
  • Kafka depends on ZooKeeper for critical tasks such as:
    1. Leader Election: ZooKeeper determines which broker will be the leader for each topic partition.
    2. Metadata Storage: ZooKeeper stores information about brokers, topics, and partitions.
    3. Configuration Management: ZooKeeper manages broker configurations and their updates.
  • In these versions, Kafka cannot function without ZooKeeper.
• Advantages of Using ZooKeeper:
  1. Decouples metadata management from Kafka brokers.
  2. Ensures a robust consensus algorithm for leader election.

When ZooKeeper Is Not Required

• Kafka Versions 2.8 and Above (KRaft Mode):
  • Kafka introduced the KRaft (Kafka Raft) protocol, which removes the dependency on ZooKeeper.
  • KRaft Mode:
    1. Kafka brokers use the Raft consensus algorithm for metadata management and leader election.
    2. One or more brokers act as the controllers, eliminating the need for an external ZooKeeper cluster.
    3. The metadata is stored directly within Kafka, simplifying deployment and operations.
• Advantages of KRaft Mode:
  1. Simplified Architecture:
     • No need for a separate ZooKeeper cluster.
     • Easier deployment and maintenance.
  2. Improved Scalability:
     • Handles larger clusters and higher metadata update rates.
  3. Higher Performance:
     • Direct communication between brokers without the latency introduced by ZooKeeper.
  4. Unified Security:
     • Only Kafka’s security mechanisms (e.g., SASL, ACLs) need to be managed.

Migration from ZooKeeper to KRaft

• Kafka 2.8 and 3.x:
  • Kafka operates in either ZooKeeper mode or KRaft mode.
  • To migrate:
    1. Upgrade Kafka to a version supporting KRaft.
    2. Gradually migrate metadata from ZooKeeper to KRaft using Kafka tools.
  • After migration, ZooKeeper can be retired.
• Kafka 4.0 and Above:
  • Kafka will operate only in KRaft mode, completely deprecating ZooKeeper.

ZooKeeper vs. KRaft: A Comparison

Key Considerations

1. Legacy Clusters:
   • If your Kafka version is below 2.8, ZooKeeper is still a hard requirement.
   • For older clusters, ensure ZooKeeper is set up with proper redundancy (e.g., a 3 or 5-node ZooKeeper ensemble).
2. New Deployments:
   • For Kafka 2.8+ and newer, it is recommended to use KRaft mode to avoid the operational overhead of managing ZooKeeper.
3. Future Proofing:
   • Since ZooKeeper will be completely deprecated in Kafka 4.0, planning a migration to KRaft mode is a good long-term strategy.

Conclusion

• ZooKeeper is not mandatory for Kafka versions 2.8 and above running in KRaft mode.
• For older versions or clusters not yet migrated, ZooKeeper remains essential.
• Moving to KRaft simplifies the architecture, improves performance, and aligns with Kafka’s future roadmap.