Apache Kafka is a distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

How Kafka Stores Partitions

Kafka stores data in topics. Topics are divided into partitions, which are essentially the fundamental unit of parallelism in Kafka. Partitions allow a topic to be parallelized by splitting the data across multiple brokers in the cluster.

Each partition in Kafka is an ordered, immutable sequence of records that is continually appended to—a structured commit log. Every record in a partition is assigned and identified by its offset, a sequential id that uniquely identifies records within the partition.

Storage on Disk

Each partition of a topic is stored on disk as a set of log files, and each log file is a simple sequence of records. Kafka appends records to these logs and serves them in the same order, ensuring immutability and sequence integrity.

The data within partitions is saved in the file system of the server that hosts the Kafka broker. Each broker typically has a configuration parameter (log.dirs) that specifies the directories where log data is stored. For fault tolerance and reliability, it's common to configure multiple log directories on different devices.

Replication of Partitions

For fault tolerance, Kafka replicates partitions across multiple brokers. This means that each partition may have one or more replicas distributed across the cluster, with one of them designated as the leader. All read and write requests go to the leader of the partition, and the leader then synchronizes the records with its replicas.

The number of replicas for a partition is set by the replication factor, which is a topic-level configuration. A higher replication factor decreases the probability of data loss but increases the overhead on the Kafka cluster.

Example of Partition Storage

Suppose you configure a Kafka broker with log.dirs=/kafka/logs1,/kafka/logs2. If there is a topic T1 with two partitions P1 and P2, the data might be distributed as follows:

• Partition P1 could be located in /kafka/logs1/T1-0
• Partition P2 could be located in /kafka/logs2/T1-1

Each folder (T1-0, T1-1) contains multiple log files that are rotated according to settings like maximum size or time-to-live.

Handling Broker Failures

If a broker fails, Kafka provides high availability of data by shifting partition leadership to one of the replicas, which would have an identical copy of the data as the original leader. This is managed automatically by Kafka's controller, which runs on one of the brokers in the cluster.

Summarizing Key Points

Below is a table summarizing the key points regarding how Kafka stores partitions:

Additional Details

• Log Compaction: Kafka supports a feature called log compaction that ensures the log contains at least the last known value for each record key. This is particularly useful for restoring state after a failure or restart.
• File Management: Kafka manages the partition logs by committing old log segments to free up space, configurable through policies based on size or time.

By managing data across multiple brokers and ensuring reliability through partition replication, Kafka provides a robust and scalable architecture for streaming data needs.