Kafka Connect
Confluent Docker
Jar Files
Classpath Configuration
Docker Container Management

Add jar to general Kafka Connect classpath in the Confluent Docker

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

In Confluent's Kafka Connect Docker images, there are two different questions people often mix together: "how do I add a connector plugin" and "how do I add a library to the worker-wide classpath." For connector JARs, the preferred mechanism is usually CONNECT_PLUGIN_PATH, while CLASSPATH is the more general but riskier escape hatch for worker-level libraries.

Prefer Plugin Isolation for Connectors

If you are adding a connector, converter, or SMT plugin, the normal approach is to place it in its own directory and point Kafka Connect at that plugin path.

A Docker Compose example looks like this:

yaml
1services:
2  connect:
3    image: confluentinc/cp-kafka-connect:latest
4    environment:
5      CONNECT_PLUGIN_PATH: "/usr/share/java,/etc/kafka-connect/plugins"
6    volumes:
7      - ./plugins:/etc/kafka-connect/plugins

Each plugin should ideally live in its own subdirectory under the mounted plugin path. That keeps dependency isolation cleaner and matches how Kafka Connect expects plugin discovery to work.

When You Actually Need General Classpath

Sometimes you truly need a library available to the worker process itself, not just inside one isolated plugin directory. In that case, you can extend the image and set CLASSPATH.

dockerfile
1FROM confluentinc/cp-kafka-connect:latest
2
3COPY ./worker-libs/*.jar /opt/connect-extra-libs/
4ENV CLASSPATH="/opt/connect-extra-libs/*"

This makes the JARs visible to the general JVM classpath for the Connect worker.

That can be useful for:

  • custom security providers
  • worker-wide libraries
  • special dependencies not resolved well through plugin isolation

But it is also easier to create classpath conflicts this way.

Why CLASSPATH Is Not the First Choice

Kafka Connect plugin isolation exists to reduce dependency collisions between connectors. If you push too much into the general classpath, you undo some of that isolation and increase the chance of:

  • version conflicts
  • class shadowing
  • surprising runtime behavior between connectors

So the practical rule is:

  • use CONNECT_PLUGIN_PATH for connector-style additions
  • use CLASSPATH only when you really need worker-wide visibility

Custom Image Versus Volume Mount

For development, a mounted volume can be fine. For production, a custom image is usually safer because it is reproducible and easier to promote through environments.

Example custom image with plugin directory:

dockerfile
1FROM confluentinc/cp-kafka-connect:latest
2
3COPY ./plugins/my-connector /etc/kafka-connect/plugins/my-connector
4ENV CONNECT_PLUGIN_PATH="/usr/share/java,/etc/kafka-connect/plugins"

This avoids depending on a host-mounted directory layout at runtime.

Verify That Connect Sees the Plugin

After the worker starts, confirm discovery through the REST API:

bash
curl http://localhost:8083/connector-plugins

If the plugin does not appear, the most common causes are:

  • wrong directory structure
  • wrong CONNECT_PLUGIN_PATH
  • missing transitive JARs
  • stale container image

For general classpath issues, the plugin may appear but fail later with ClassNotFoundException or dependency conflicts in the worker logs.

That distinction matters because discovery failures and runtime linkage failures have different fixes. One is usually a path problem, while the other is usually a dependency or classloader problem.

Common Pitfalls

  • Using CLASSPATH for connector plugins when plugin isolation is the better mechanism.
  • Dumping multiple unrelated connector JARs into one flat directory and making dependency conflicts harder to diagnose.
  • Forgetting to rebuild the Docker image after adding new JARs.
  • Assuming plugin discovery and worker classpath are the same thing.
  • Ignoring /connector-plugins output and debugging blindly from connector creation failures.

Summary

  • In Confluent Docker, connector plugins should usually be added through CONNECT_PLUGIN_PATH.
  • Use CLASSPATH only when you need worker-wide library visibility.
  • Prefer custom images over ad hoc mounted JAR directories for reproducible deployments.
  • Keep plugins isolated in their own directories to reduce dependency conflicts.
  • Verify plugin discovery through the Kafka Connect REST API after startup.

Course illustration
Course illustration

All Rights Reserved.