Kafka
Partitions
Offset Number
Data Management
Distributed Systems

Kafka, will different partitions have the same offset number

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines and applications. It primarily functions by storing, reading, and analyzing streaming data. Kafka is designed around a distributed commit log and offers features like high-throughput, fault-tolerance, and scalability. Kafka’s core abstraction is a topic, which can be divided into multiple partitions for parallel processing.

Understanding Kafka Partitions and Offsets

Partitions

A Kafka topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it. For scalability, topics are split into partitions. Each partition is an ordered, immutable sequence of records that is continually appended to—a commit log. Each record in a partition is assigned and identified by its unique offset.

Partitioning in Kafka serves several purposes:

  • Parallelism: By dividing the data among multiple nodes, partitions allow the data for a topic to scale beyond a single server.
  • Fault Tolerance: Partitions can be replicated across multiple nodes to ensure that data is not lost if a node fails.

Offsets

An offset is a unique identifier for a record within a partition. It denotes the position of each record within the immutable sequence of the partition. The key point here is that offsets are unique per partition. This means that two records from different partitions of the same topic can have the same offset.

For example, in a topic with two partitions, Partition 0 and Partition 1, both can independently have records with offsets 0, 1, 2, and so forth. Here’s a basic representation:

TopicPartitionOffsetMessage
TopicA00Hello
TopicA01World
TopicA10Test
TopicA11Data

Notice how both Partition 0 and Partition 1 have the offsets 0 and 1, each referring to completely different messages.

Technical Implications of Partitions and Offsets

  1. Consumer Offsets: Kafka uses a distributed storage mechanism for consumer offsets (the position of a consumer in a partition). This enables it to resume reading from where it left off, even in the event of a restart.
  2. Ordering Guarantees: Kafka guarantees the order of messages only within a specific partition, not across partitions.

Example of Topic with Multiple Partitions

Consider a topic "Orders" designed to receive data related to orders placed in an e-commerce platform. Assuming this topic is divided into two partitions:

  • Partition 0 might store records with offsets for order IDs in sequential manner but only for even IDs.
  • Partition 1 stores records with offsets for order IDs but only for odd IDs.

Even though both partitions are part of the same topic, they maintain separate offset sequences.

Summary Table

ConceptDescription
TopicA stream of messages of a similar type
PartitionA split of a topic to achieve parallelism
OffsetUnique identifier of a record within a partition

Conclusion

In summary, Kafka guarantees unique offsets within a partition but not across different partitions under the same topic. This design allows Kafka to achieve high levels of parallelism and scalability. By understanding how partitions and offsets work, developers and architects can design more efficient streaming applications using Apache Kafka.


Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.