Introduction

Kafka events can be written directly to a database or first landed in S3 or HDFS before downstream loading. The right pattern depends on latency requirements, replay strategy, transformation complexity, and storage economics. Choosing well requires separating operational goals from implementation convenience.

Direct Kafka to Database Pattern

Direct sinks are common when applications need low-latency queryable data.

Typical flow:

1. producer writes events to Kafka topic.
2. sink connector or consumer service reads topic.
3. data is upserted into operational database.

Kafka Connect sink example shape:

1name=orders-jdbc-sink
2connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
3tasks.max=2
4topics=orders
5connection.url=jdbc:postgresql://db.internal:5432/appdb
6connection.user=app_user
7connection.password=change_me
8insert.mode=upsert
9pk.mode=record_key
10pk.fields=order_id
11auto.create=false
12auto.evolve=false

Strengths:

• near real-time availability.
• simpler path for operational dashboards.
• fewer moving components.

Tradeoff:

• database write pressure can become bottleneck at high throughput.

Kafka to S3 or HDFS First Pattern

Landing raw events in object or distributed file storage adds a durable replay and analytics layer.

Flow:

1. consume Kafka topic into partitioned files.
2. store immutable raw events by time and topic.
3. run batch or streaming transforms into serving database.

Benefits:

• cheaper long-term storage for large event volume.
• replayability for backfills and schema migration.
• decouples ingestion spikes from database capacity.

Example S3 connector shape:

1name=orders-s3-sink
2connector.class=io.confluent.connect.s3.S3SinkConnector
3topics=orders
4s3.bucket.name=company-raw-events
5flush.size=10000
6format.class=io.confluent.connect.s3.format.json.JsonFormat
7partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner
8path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH
9locale=en
10timezone=UTC

This layout supports scalable downstream processing with Spark, Flink, or warehouse loaders.

Latency Versus Flexibility Decision

Choose direct database sink when:

• consumers need immediate queryability.
• transformation logic is minimal.
• event volume fits database write profile.

Choose S3 or HDFS staging when:

• replay and audit requirements are strong.
• complex transformations are required.
• throughput bursts exceed stable DB ingestion capacity.

Many teams adopt a dual path:

• real-time subset to operational DB.
• full raw stream to data lake for history and recovery.

Idempotency and Exactly-Once Reality

Regardless of architecture, duplicates happen during retries and failures unless sink layer is idempotent.

Key safeguards:

• stable event keys.
• upsert semantics in database.
• deduplication rules in downstream batch loads.
• checkpointing tied to sink commits.

Do not assume connector-level exactly-once covers every storage and network failure mode. Validate with failure tests.

Cost and Operational Complexity

Direct-to-database reduces components but can increase DB cost and contention. Staging through S3 or HDFS adds system complexity but often lowers storage cost and improves historical analytics capabilities.

Operationally, evaluate:

• connector maintenance overhead.
• schema evolution handling.
• replay speed and procedure.
• monitoring depth for lag, error rate, and sink latency.

A small architecture that cannot replay data can become expensive during incident recovery.

Schema Evolution Strategy

Whichever path you choose, schema evolution must be planned up front. Use schema registry and compatibility rules so producers and sinks evolve safely over time.

Recommended pattern:

1. enforce backward-compatible event schema changes.
2. version sink transformations explicitly.
3. keep raw immutable data for replay when mapping logic changes.

This avoids emergency data backfills when downstream contracts change unexpectedly.

Common Pitfalls

• Writing all events directly to OLTP database without capacity headroom.
• Skipping immutable raw storage and losing replay ability.
• Treating connector retries as full data-correctness guarantee.
• Ignoring schema evolution impacts on long-lived pipelines.
• Choosing architecture by tool familiarity instead of workload requirements.

Summary

• Direct Kafka-to-DB is best for low-latency operational use cases.
• S3 or HDFS staging is best for replayability, scale, and analytics pipelines.
• Hybrid architectures often provide the strongest practical balance.
• Idempotent sink design is essential in both patterns.
• Make decisions using latency, cost, and recovery requirements together.