Introduction

Kafka broker settings can come from more than one place, so conflicts do happen. A property may be defined in server.properties, overridden dynamically for the whole cluster, and overridden again for one specific broker. When that happens, Kafka follows a clear precedence order, and knowing that order is essential when you are debugging unexpected broker behavior.

The Precedence Order

For broker configurations that support dynamic updates, the effective value is chosen in this order:

1. dynamic per-broker configuration
2. dynamic cluster-wide default configuration
3. static broker configuration in server.properties
4. Kafka’s built-in default value

That means a value attached directly to broker 1 wins over a cluster-wide override, and both dynamic values win over whatever is written in the static config file.

Static Versus Dynamic Configuration

The static layer is the broker’s local configuration file, usually config/server.properties. This is the traditional place to define properties such as log directories, listeners, and thread counts.

Example:

log.cleaner.threads=1

Kafka also supports dynamic broker configuration for many properties. Those values are stored in Kafka’s metadata layer and can be altered with kafka-configs.sh without editing the file directly.

A per-broker override looks like this:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-name 1 \
5  --alter \
6  --add-config log.cleaner.threads=3

A cluster-wide default override looks like this:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-default \
5  --alter \
6  --add-config log.cleaner.threads=2

With those values in place:

• broker 1 uses 3
• other brokers use 2
• the static server.properties value of 1 is ignored until the dynamic overrides are removed

A Concrete Conflict Example

Suppose you have this in server.properties on every broker:

num.network.threads=3

Then you apply a cluster-wide dynamic default:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-default \
5  --alter \
6  --add-config num.network.threads=5

Now every broker that does not have its own broker-specific override uses 5.

If you then apply a per-broker override on broker 2:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-name 2 \
5  --alter \
6  --add-config num.network.threads=8

The result becomes:

• broker 2 uses 8
• all other brokers use 5
• none of them currently use the static 3

If you delete the per-broker override for broker 2, it falls back to the cluster-wide dynamic value of 5. If you also delete the cluster-wide override, brokers fall back to server.properties, which is 3.

How to Inspect What Is Active

When you are unsure why a broker is behaving a certain way, inspect the dynamic config directly.

Describe a specific broker:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-name 2 \
5  --describe

Describe the cluster-wide defaults:

1bin/kafka-configs.sh \
2  --bootstrap-server localhost:9092 \
3  --entity-type brokers \
4  --entity-default \
5  --describe

Those commands are often more informative than staring at server.properties, because the file may no longer be the winning source.

Important Limitation: Not Every Property Is Dynamically Updatable

Precedence does not mean every property can be changed live. Kafka marks broker settings with update modes such as read-only, per-broker, or cluster-wide. Read-only settings still require a restart, even if you wish they were dynamic.

So the full rule is:

• if a property supports dynamic configuration, the precedence order above applies
• if it does not, the file-based static value and restart behavior still govern the result

Common Pitfalls

The biggest mistake is editing server.properties and expecting the broker to pick up the change while a dynamic override is still present. It will not. The dynamic value continues to win until it is deleted.

Another common problem is checking only one layer. If you inspect the file but never describe dynamic broker configs, you can draw the wrong conclusion about the active setting.

Older examples also confuse people because they rely on ZooKeeper-era commands. For current Kafka setups, prefer the --bootstrap-server form unless you are working in an older environment on purpose.

Finally, do not assume a property is dynamically updatable just because another one is. Kafka documents update modes per setting, and that difference matters operationally.

Summary

• Kafka broker config precedence is dynamic per-broker, then dynamic cluster default, then server.properties, then built-in default.
• A per-broker dynamic override beats both a cluster-wide dynamic override and the static file.
• Removing a higher-precedence override causes Kafka to fall back to the next layer.
• Use kafka-configs.sh --describe to inspect active dynamic settings.
• Not every broker property supports live dynamic updates, so always check the update mode.