Introduction

Kafka can run on Kubernetes in an Istio-managed environment, but it needs more deliberate network design than a normal HTTP microservice. Kafka is a TCP-based system with broker-specific addresses, long-lived connections, and metadata-driven routing, so the usual service-mesh assumptions do not automatically fit.

Why Kafka Is Different

Kafka clients do not connect to one stable service name forever. A typical client flow is:

1. connect to a bootstrap broker
2. fetch cluster metadata
3. learn which brokers lead which partitions
4. open direct connections to those broker addresses

That means the broker addresses returned in metadata must be reachable from the client’s real network context.

This is the central design rule for Kafka on Kubernetes with Istio: the advertised broker identities must remain valid after all service-mesh and Kubernetes networking layers are applied.

Listener Design Comes First

Before tuning Istio policy, get Kafka listener design right. In Kubernetes, brokers usually need stable per-broker identities, often through a StatefulSet plus a headless service.

A simplified pattern:

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-0.kafka-headless.default.svc.cluster.local:9092
inter.broker.listener.name=PLAINTEXT

Each broker must advertise an address that intended clients can actually resolve and route to.

If this is wrong, no amount of mesh tuning will fix the deployment. Metadata will point clients toward unusable addresses and produce the familiar “bootstrap worked, but actual produce/consume failed” pattern.

Decide Deliberately About Sidecars

One of the first architecture decisions is whether Kafka brokers should run with Istio sidecars at all.

Common strategies are:

• exclude brokers from sidecar injection and keep Kafka networking close to native Kubernetes behavior
• keep brokers in the mesh, but configure Istio explicitly for Kafka’s TCP traffic and broker identity requirements

Many teams choose to exclude brokers from sidecars because Kafka is sensitive to connection handling and because Istio defaults are usually designed first for HTTP workloads.

That is not a universal law. It is just a common operational simplification. If you do keep sidecars on brokers, treat it as a deliberate design choice and test it thoroughly.

Treat It as TCP, Not HTTP

Kafka is not an HTTP routing problem. Istio configuration therefore has to respect plain TCP behavior.

A simplified DestinationRule example:

1apiVersion: networking.istio.io/v1beta1
2kind: DestinationRule
3metadata:
4  name: kafka-dr
5  namespace: default
6spec:
7  host: kafka-headless.default.svc.cluster.local
8  trafficPolicy:
9    tls:
10      mode: DISABLE

The exact policy depends on your security model, but the larger point is that Kafka networking should be reasoned about as TCP transport, not as HTTP request routing.

Understand the Security Layers

When Kafka and Istio are combined, there may be multiple security layers:

• Istio mTLS between workloads
• Kafka TLS for broker and client communication
• Kafka SASL for client identity
• Kafka ACLs for topic authorization

You need a clear plan for which layer does what.

A workable separation is often:

• Istio handles pod-to-pod transport policy
• Kafka handles client identity and topic authorization

The dangerous state is an accidental half-configured mix where the mesh expects one transport pattern and Kafka expects another. That kind of mismatch often produces connection resets and metadata failures that look mysterious until you inspect both layers together.

Test From Real Client Contexts

Do not test only from a broker pod or from a privileged admin shell. Test from the same namespace and workload style your real producers and consumers use.

Example:

1kubectl exec -it deploy/test-client -- \
2  kafka-topics.sh --bootstrap-server kafka-0.kafka-headless.default.svc.cluster.local:9092 --list
3
4kubectl exec -it deploy/test-client -- \
5  kafka-console-producer.sh --broker-list kafka-0.kafka-headless.default.svc.cluster.local:9092 --topic smoke

Bootstrap success is not enough. Produce and consume must work after metadata exchange, because that is the stage where incorrect advertised endpoints usually break clients.

Common Pitfalls

The biggest mistake is advertising broker endpoints that are only valid from inside the broker pod or only from one namespace.

Another issue is treating Kafka as if ordinary Istio HTTP patterns automatically apply. Kafka is a TCP protocol with broker-aware client routing.

People also often validate only bootstrap connectivity and stop there. Many real failures appear only after the client receives metadata and tries broker-specific endpoints.

Finally, do not run brokers with sidecars by accident. Whether brokers participate in the mesh should be an explicit architecture decision.

Summary

• Kafka on Kubernetes with Istio is mainly a broker-identity and TCP-transport problem.
• Stable, correct advertised.listeners values matter more than most higher-level mesh details.
• Decide explicitly whether brokers should run with or without sidecars.
• Configure Istio with Kafka’s TCP behavior in mind, not HTTP assumptions.
• Test full produce-and-consume flows from real client contexts, not just bootstrap connectivity.