Kafka Connect is a cluster of JVM processes that run other people's code on your behalf, and almost every operational surprise it produces comes from not having a clear picture of which process is doing what. So the object model first, precisely.

The four objects

Each one is a different kind of thing, and conflating any two of them is the source of most confusion when reading Connect's documentation or its REST responses.

A worker is a JVM process. It is what you deploy, what has a heap, what appears in your orchestrator, and what exposes the REST API on port 8083.

A connector is a configuration plus a small coordinating object. It does not move data. Its entire job is to inspect its configuration and the external system and decide how the work should be divided.

A task is the unit that actually moves data. A source task polls an external system and returns records; a sink task receives records and writes them. Tasks are what get distributed across workers.

And a plugin is the jar or directory containing the connector's classes, discovered from plugin.path at worker startup. A plugin that is not on the path does not exist as far as the REST API is concerned, which produces a confusing class-not-found error at connector creation time rather than at startup.

How the work is divided

The division happens once, in the connector, and understanding it explains most of the throughput questions people ask.

When a connector is created or reconfigured, Connect calls taskConfigs(maxTasks) on it, passing the tasks.max from the configuration. The connector returns a list of task configurations, and the length of that list is how many tasks actually run.

The crucial detail is that tasks.max is a ceiling, not a target. A connector returns fewer configurations whenever it cannot divide the work further, and that is extremely common: a JDBC source in bulk mode on a single table returns one, a Debezium source for MySQL returns exactly one because a single binlog cannot be read in parallel, and a file source returns one per file.

So raising tasks.max from 1 to 12 on a connector that returns one task changes nothing at all. The only way to know is to create it and read back the task list from GET /connectors/<name>/status, which is the first thing to do when a source connector will not go faster.

Sink tasks are consumers, which fixes their ceiling differently

Sink connectors divide according to Kafka rather than according to the connector, which is a different rule and a more predictable one.

Each sink task is a Kafka consumer in a group named connect-<connector-name>. So the task count is bounded by the partition count of the topics it consumes, exactly as for any consumer group: 6 partitions and tasks.max=12 gives 12 tasks of which 6 are idle.

This also means a sink connector's scaling, rebalancing and lag behave exactly like a consumer group's, and all the consumer tooling applies. kafka-consumer-groups.sh --describe --group connect-my-sink is the most useful command for a slow sink connector, and it is routinely overlooked because people look for Connect-specific tooling first.

Standalone and distributed

Two modes, and only one of them should ever be in production.

Standalone runs one worker with connector configurations from properties files and offsets in a local file given by offset.storage.file.filename. It is for development and for genuinely single-machine jobs. Losing that file loses the position.

Distributed runs multiple workers that coordinate through Kafka itself and store everything in three internal topics. Connectors are created through the REST API, work is assigned across workers, and a worker's death causes a rebalance rather than an outage.

A single worker in distributed mode is a perfectly reasonable deployment and is strictly better than standalone, because the state is in Kafka and a second worker can be added later without migrating anything. There is no good reason to choose standalone for anything that is expected to keep running.

The group membership, which is not what people expect

Workers form a consumer group of their own, and knowing this makes the cluster's behaviour legible.

Every worker with the same group.id is a member of one group, using the same group-membership protocol as a consumer group. One worker is elected leader and is responsible for computing the assignment of connectors and tasks to workers.

So a worker joining or leaving triggers a worker rebalance, during which assignments are recomputed. With the older eager protocol every task in the cluster stopped during that rebalance, which is why Connect clusters had a reputation for stopping everything when one worker restarted.

connect.protocol=sessioned, the default since 2.4 and the implementation of incremental cooperative rebalancing, changes that: only the assignments that actually need to move are revoked, and scheduled.rebalance.max.delay.ms at 300,000 holds a departed worker's tasks unassigned for up to five minutes in case it comes back, which avoids a double rebalance on a rolling restart.

What this means for a rolling restart

The practical consequence, because it is the operation performed most often and the one most likely to be done wrongly.

With scheduled.rebalance.max.delay.ms at its default, restarting a worker leaves its tasks unassigned for up to five minutes rather than immediately reassigning them. That is deliberate and correct for a restart, and it looks like an outage to anyone watching task counts.

So a rolling restart should allow each worker to rejoin before moving to the next, and the health check should be the connector's task states through the REST API rather than the worker's own liveness, because a worker that is up with no assignments looks healthy and is doing nothing.

Restarting the leader is no more significant than restarting any other worker, since the leader election is part of the group protocol and takes a rebalance, not an operator action.

Reading a status response, field by field

One API call answers most questions about a connector, and the fields are worth knowing individually because each one lies in a different way.

GET /connectors/<name>/status returns a connector object with a state and the worker_id of the worker running the connector object, and a tasks array with one entry per task, each carrying an id, a state, a worker_id and, for a failed task, a trace holding the stack trace.

The connector-level state is the least informative field in the response, because the connector object does almost nothing. A connector in RUNNING with every task in FAILED is a connector that is moving no data at all, and that is the single most common Connect situation that gets misread from a dashboard built on the connector state alone.

The task worker_id values tell you the distribution, which matters when one worker is saturated: four tasks all on the same worker is a legitimate assignment under incremental cooperative rebalancing, because that protocol minimises movement rather than balancing perfectly, and the fix is a rebalance rather than a configuration change.

And the trace is the whole diagnosis for a failed task, because Connect writes the original exception there. It is also the only place it appears if the worker's logs have rotated, which is a good reason to capture the status response rather than only alerting on it.

Plugin isolation, and the error it produces

A classloader detail that is invisible until it is the entire problem.

Connect loads each plugin in its own classloader, scanned from the directories listed in plugin.path, which is a comma-separated list of directories each containing plugin directories or uber-jars. The isolation exists because two connectors routinely depend on incompatible versions of the same library, and without it one connector's Jackson version breaks another's.

The isolation is not total. Classes in the Connect API itself, and a short list of packages including org.apache.kafka.connect and the JDK's own, are loaded from the parent classloader so that a plugin's SourceRecord is the same class as the framework's. A plugin that bundles its own copy of the Connect API fails in confusing ways for exactly this reason.

The practical failure is a connector creation that returns a 500 with a class-not-found for the connector class, when the jar is sitting in a directory that is inside plugin.path rather than being a plugin directory under one. The distinction is one directory level and it is the commonest Connect installation mistake.

GET /connector-plugins lists what the worker actually found, which answers the question definitively and should be the first call after any plugin installation rather than attempting to create the connector and reading the error.

Why the asymmetry is the right design

Worth stating because it makes the rest of the framework predictable.

A source's achievable parallelism is a property of the external system: a binlog is serial, a table can be split by a key range, a message queue can be consumed by many clients. Connect knows nothing about any of those, so it has to delegate the decision to the connector, which is the only component that understands the source.

A sink's achievable parallelism is a property of Kafka, which Connect understands completely, so it applies the standard consumer-group rule and needs no help from the connector.

So the two halves of Connect have different scaling rules because they have different sources of truth about what parallelism is possible, and knowing which half you are looking at tells you which question to ask: for a source, ask the connector how it divides work; for a sink, ask Kafka how many partitions there are.