Kafka Streams is a library, not a cluster. There is no Streams server to deploy, no scheduler, and no master. An application is an ordinary JVM process that embeds consumers, producers and a local state store, and everything about its scaling and failure behaviour follows from that one fact. The unit that makes it legible is the topology.

What the topology actually is

A directed graph, and it is worth being precise about what the nodes are because the DSL hides them.

A topology is a graph of processor nodes connected by streams of records. A source node reads from a Kafka topic, a processor node transforms, and a sink node writes to a topic. The DSL's filter, map, groupBy and join all compile down to processor nodes, and several of them compile to more than one.

topology.describe() prints the compiled graph, and printing it is the single most useful thing to do with a Streams application you did not write. It names every node, shows which stores each one accesses, and, critically, shows where the topology has been split.

That output is also the only place some things are visible at all. A groupBy that re-keys records creates an internal topic whose existence does not appear anywhere in the application's source, and a developer reading the DSL code has no way to know how many Kafka round trips their three-line pipeline performs.

Sub-topologies, which are where the round trips live

The split rule, because it determines the application's real shape.

A topology is divided into sub-topologies at every point where records must be redistributed across partitions. That happens when a key changes, because downstream aggregation requires all records for a key to be in one partition, and a key change means the record is now in the wrong one.

So groupByKey on a stream whose key is unchanged stays in the same sub-topology and costs nothing. groupBy with a new key splits the topology, writes every record to an internal repartition topic, and reads it back. That is a full produce-and-consume round trip through the brokers, for every record.

The same split happens for a stream-stream join on a re-keyed stream, and for through or repartition called explicitly. Each split is a boundary at which records leave the process, hit disk on a broker, and come back.

A pipeline that reads a topic, re-keys it, aggregates, re-keys the result and aggregates again is therefore three sub-topologies and two repartition round trips. The DSL code is five lines, the network cost is triple, and the only honest measure of a topology's weight is its sub-topology count rather than its line count.

Tasks, which are the unit of parallelism

The definition, and it is the one that explains every scaling question.

A task is one sub-topology instantiated for one partition number. If a sub-topology reads a topic with 12 partitions, there are 12 tasks for it, numbered 0_0 through 0_11, where the first number is the sub-topology id and the second the partition.

The total task count is therefore the sum, across sub-topologies, of the maximum partition count among that sub-topology's input topics. A topology with two sub-topologies reading 12-partition topics has 24 tasks, and that number is fixed by the topics rather than by anything in the application's configuration.

Tasks are the unit that gets assigned to instances, so the task count is the hard ceiling on parallelism. Running 30 instances of an application with 24 tasks leaves 6 instances holding nothing, and the only way to raise the ceiling is to add partitions to the input topics, which is a Kafka change rather than an application one.

This is why partition count is a capacity decision made early and regretted later. Adding partitions to a topic feeding a stateful aggregation also changes which key hashes to which partition, which invalidates the existing state entirely.

Threads, and what num.stream.threads really controls

One setting, commonly misunderstood as the parallelism knob.

num.stream.threads defaults to 1 and sets how many stream threads one instance runs. Each thread owns a set of tasks and processes them in a loop, polling its consumer, running records through the topology and committing.

So the total parallelism is instances times threads, bounded by the task count. Ten instances at one thread each and one instance at ten threads give the same task distribution and very different failure behaviour, which is the real reason to prefer instances: a thread's death takes its tasks down, and a process's death takes all of its threads' tasks down.

Each thread has its own consumer and its own producer, which means raising num.stream.threads multiplies the client count against the brokers. Twenty instances at four threads is 80 consumers in one group, and group sizes in that range make rebalances noticeably slower.

A thread that throws an uncaught exception is replaced rather than killed from 2.8 onward, under the StreamsUncaughtExceptionHandler's REPLACE_THREAD option, which is a better default than the old behaviour of leaving the application running with fewer threads and no indication.

The assignor, which is not the consumer assignor

A substitution worth knowing about, because it changes what the group protocol is doing.

Streams replaces the consumer's partition assignor with StreamsPartitionAssignor, which assigns tasks rather than partitions and then derives the partition assignment from the tasks. It has to, because a task may read several topics and all of them must go to the same instance.

The assignor is also what implements stickiness for state: it prefers to give a task back to the instance that already has that task's local state on disk, because the alternative is restoring the state from its changelog. That preference is the difference between a rebalance costing milliseconds and costing minutes.

It runs on the group leader, so the leader of a Streams group does meaningfully more work than other members, and the computation is non-trivial for a large task count.

And it embeds Streams-specific metadata in the group's subscription, which is why mixing Streams versions with incompatible assignor protocol versions in one group fails rather than degrading. Version upgrades across that boundary need the documented two-step upgrade.from procedure.

Reading an application's shape before changing it

Five questions, because they are the ones whose answers determine what any change will cost.

How many sub-topologies does topology.describe() show, and therefore how many repartition round trips does each record make?

How many tasks exist, which is the ceiling on parallelism and is set by the input topics' partition counts rather than by configuration?

Which tasks are stateful, since those are the ones whose reassignment costs a restoration and the ones that make a rebalance expensive?

What is num.stream.threads times the instance count, and does that product exceed the task count, which would mean instances holding nothing?

And what are the repartition topics named, since they are created automatically with a name derived from the application.id and a generated node name, and they are the topics that accumulate data nobody is watching.

Reading the split out of the code

Because the DSL hides it, and a small example makes the rule concrete.

1KStream<String, Order> orders = builder.stream("orders");
2
3// same sub-topology: the key is unchanged, so no redistribution is needed
4orders.filter((k, v) -> v.total() > 0)
5      .groupByKey()
6      .count(Materialized.as("orders-per-customer"));
7
8// new sub-topology: groupBy re-keys, so every record goes through
9// an internal repartition topic and comes back
10orders.groupBy((k, v) -> v.region(), Grouped.with(Serdes.String(), orderSerde))
11      .count(Materialized.as("orders-per-region"));

The two branches look symmetric in the source and are not symmetric at all in cost. The first stays in place because groupByKey asserts that the existing key is the grouping key, so the records are already in the right partitions. The second changes the key to the region, which means a record sitting in partition 7 because of its customer id must move to whichever partition the region hashes to, and the only way Streams can move a record between partitions is to write it to a topic.

The naming in Materialized.as is not cosmetic either. Without it the store and its changelog take a generated name derived from the node's position in the topology, and inserting an operation above it later shifts that position. Naming is covered properly in the operating section, but the habit starts here.

Two numbers to compute before deploying

Both are arithmetic and both are routinely skipped.

The first is the task count, which is the sum over sub-topologies of the largest partition count among that sub-topology's inputs. For the example above, with orders at 12 partitions, the first sub-topology contributes 12 and the repartitioned one contributes another 12, so 24 tasks exist and 24 is the parallelism ceiling.

The second is the record multiplier, which is how many broker writes one input record causes. In the example each input record is read once, counted in place, and then written to a repartition topic and read back, so one input record produces one repartition write and one repartition read on top of two changelog writes. Four broker interactions for a pipeline that reads as two counts.

That multiplier is the number to take to a capacity conversation, because it is what the brokers actually see. A team sizing a cluster from the input topic's rate alone will be out by whatever the multiplier is, and for a topology with three repartitions it is not a small factor.