Introduction

Running Kafka connectors in a Helm-based Kubernetes deployment is mostly about correct Kafka Connect operations. Connector JSON alone is not enough if plugin binaries, internal topics, networking, or secrets are misconfigured. A stable setup requires clear worker configuration, controlled plugin delivery, and post-deployment verification.

Understand the Deployment Model

A connector does not run in Kafka brokers. It runs in Kafka Connect workers. In a Helm deployment, workers are pods managed by Kubernetes resources.

You need all of these aligned:

• Kafka brokers reachable from Connect workers.
• Internal Connect topics for config, offsets, and status.
• Connector plugin classes available in worker plugin path.
• Network access from workers to source or sink systems.

If one layer is missing, connector creation may succeed while tasks fail repeatedly.

Configure Connect Worker Settings in Helm Values

Define critical worker settings explicitly instead of relying on defaults.

1connect:
2  enabled: true
3  configurationOverrides:
4    config:
5      - "bootstrap.servers=kafka:9092"
6      - "group.id=connect-cluster"
7      - "config.storage.topic=connect-configs"
8      - "offset.storage.topic=connect-offsets"
9      - "status.storage.topic=connect-status"
10      - "key.converter=org.apache.kafka.connect.json.JsonConverter"
11      - "value.converter=org.apache.kafka.connect.json.JsonConverter"
12      - "plugin.path=/usr/share/java,/etc/kafka-connect/plugins"

Apply changes:

helm upgrade --install kafka-stack confluentinc/cp-helm-charts -f values.yaml

Ensure topic replication and partition settings are appropriate for your cluster size.

Deliver Connector Plugins Properly

Connector plugin class availability is a common failure point. The safest production pattern is a custom image.

FROM confluentinc/cp-kafka-connect:7.6.0
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:latest

Push this image and reference it in values.

connect:
  image: my-registry/kafka-connect-jdbc:1.0.0

At runtime, verify plugin registration from the Connect API.

kubectl port-forward svc/kafka-connect 8083:8083
curl http://localhost:8083/connector-plugins

If the expected class is missing, connector creation will fail with class loading errors.

Create Connector Through REST API

Once workers are healthy and plugin appears, create connector config.

1curl -X POST http://localhost:8083/connectors \
2  -H "Content-Type: application/json" \
3  -d '{
4    "name": "jdbc-source-orders",
5    "config": {
6      "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
7      "tasks.max": "1",
8      "connection.url": "jdbc:postgresql://postgres:5432/app",
9      "connection.user": "app",
10      "connection.password": "secret",
11      "mode": "incrementing",
12      "incrementing.column.name": "id",
13      "topic.prefix": "orders-"
14    }
15  }'

Check connector and task state:

curl http://localhost:8083/connectors/jdbc-source-orders/status

A RUNNING status is necessary but not final proof of data movement.

Use Kubernetes Secrets for Sensitive Values

Avoid plain credentials in checked-in connector JSON. Store sensitive data in Kubernetes Secrets and inject through environment or config provider patterns.

kubectl create secret generic jdbc-creds \
  --from-literal=username=app \
  --from-literal=password='StrongPass!2026'

Then configure Connect pod environment and reference those values according to your connector and platform capabilities.

Validate End-to-End Behavior

Use a layered validation checklist:

1. Connect pod ready and healthy.
2. Plugin class visible from connector-plugins endpoint.
3. Connector status shows running tasks.
4. Destination topic receives records or sink endpoint receives writes.
5. Offset progression changes over time.

Operational commands:

kubectl get pods -l app=kafka-connect
kubectl logs deployment/kafka-connect
kubectl describe pod <connect-pod-name>

This avoids false confidence from status-only checks.

Updating Connectors Safely

For configuration changes, use PUT on the connector config endpoint. For plugin upgrades, roll worker pods so classpath reloads cleanly.

Deployment guidance:

• Version connector definitions in Git.
• Apply via CI to avoid drift.
• Pause critical connectors during risky upgrades.
• Resume and validate offset continuity after rollout.

Controlled updates reduce duplicate ingestion and silent data gaps.

Common Pitfalls

• Creating connectors before plugin classes are installed. Fix: verify connector-plugins output first.
• Using wrong broker DNS names in bootstrap.servers. Fix: test in-cluster connectivity from Connect pod.
• Leaving internal topics under-provisioned. Fix: configure topic durability according to environment.
• Committing credentials in connector JSON. Fix: move secrets into Kubernetes Secret management.
• Treating RUNNING as complete success. Fix: verify throughput and offsets, not only task state.

Summary

• Kafka connectors on Helm require worker, plugin, network, and topic alignment.
• Install plugins in worker images or mounted plugin paths before connector creation.
• Use REST API for connector lifecycle and status checks.
• Manage secrets outside connector payload files.
• Validate real data flow and offsets to confirm production readiness.