Introduction

The usual production answer for loading a Kafka topic into HDFS is Kafka Connect with an HDFS sink connector. That gives you offset management, retries, batching, and scalable operation without writing your own consumer. You can move data with a custom script or older tools such as Flume, but Kafka Connect is generally the cleanest option when the goal is reliable continuous ingestion.

Why Kafka Connect Is the Best Default

A Kafka consumer that writes to HDFS sounds simple at first, but production requirements quickly grow:

• offset tracking
• retries and failure handling
• partitioned file layout
• batching and small-file control
• schema and format management

Kafka Connect solves those operational problems for you. The HDFS sink connector reads records from Kafka topics and writes them to HDFS directories using configurable formats and partitioning strategies.

Basic Connector Configuration

An HDFS sink connector configuration looks roughly like this:

1{
2  "name": "orders-hdfs-sink",
3  "config": {
4    "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
5    "tasks.max": "2",
6    "topics": "orders",
7    "hdfs.url": "hdfs://namenode:8020",
8    "flush.size": "1000",
9    "topics.dir": "kafka",
10    "format.class": "io.confluent.connect.hdfs.parquet.ParquetFormat"
11  }
12}

Important fields:

• 'topics chooses the Kafka topic'
• 'hdfs.url points to the HDFS namenode'
• 'flush.size controls how many records are written before a file flush'
• 'format.class chooses the output format'

You post this configuration to the Kafka Connect REST API:

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  --data @hdfs-sink.json

That creates a managed connector instead of an ad hoc export script.

Think About File Layout Early

Writing every message into one flat folder is rarely good enough. Most HDFS pipelines need partitioned output for downstream engines such as Hive, Spark, or Trino.

A time-based partitioning setup might look like this:

1{
2  "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
3  "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
4  "partition.duration.ms": "3600000",
5  "timezone": "UTC",
6  "locale": "en"
7}

That produces HDFS paths grouped by hour, which is a common pattern for analytics workloads.

Output Format Matters

Plain text can work for quick debugging, but production pipelines usually prefer structured formats such as Avro or Parquet.

Why format choice matters:

• Parquet is efficient for analytics
• Avro works well with schema-based pipelines
• plain text is easy to inspect but often less efficient

Choose the format based on downstream consumers, not just the easiest first test.

Alternatives When Connect Is Not Available

If Kafka Connect is not an option, you can still read from Kafka and write to HDFS with a custom consumer. For example, a Python or Java process can poll Kafka and append to HDFS through an HDFS client library.

That can work for small pipelines, but you now own:

• offset commits
• exactly-once or at-least-once semantics
• file rolling
• retries
• monitoring

Older stacks may also use Flume, but in new Kafka-centric deployments, Connect is usually the more natural fit.

Common Pitfalls

The most common problem is forgetting that the HDFS sink connector is not part of a plain Kafka broker install. The connector plugin must be installed on the Kafka Connect worker and available on its plugin path.

Another issue is small files. If flush.size is too low or partitioning is too fine-grained, HDFS fills with tiny files that hurt downstream performance.

Security is also easy to underestimate. Real clusters may require Kerberos, TLS, or both, and the connector worker must be configured to authenticate correctly to both Kafka and HDFS.

Finally, remember that Kafka Connect sink delivery is typically at-least-once. If the pipeline retries after a failure, duplicates can appear unless downstream consumers or formats are designed to tolerate them.

Summary

• Kafka Connect with an HDFS sink connector is the standard production path from Kafka to HDFS.
• Configure the connector with topic, HDFS URL, batching, and output format.
• Plan HDFS partitioning and file layout before sending real traffic.
• Make sure the connector plugin is actually installed on the Connect worker.
• Watch for small files, authentication issues, and at-least-once duplicate behavior.