Introduction

Kafka can support chat-style messaging, but it does not provide inboxes, presence, or delivery receipts as first-class features. To use it for one-to-one and group messaging, you design those semantics yourself with topic structure, partition keys, consumer groups, and downstream storage.

Start With Kafka's Native Guarantees

Kafka gives you an append-only log, ordering within a partition, durable retention, and scalable fan-out through consumer groups. That is a strong foundation for messaging systems, but it is not the same thing as a ready-made chat broker.

The practical consequence is that you should treat Kafka as the transport and event history, while application services handle:

• recipient resolution
• inbox materialization
• unread counts
• read and delivery state
• notification fan-out

If you try to model Kafka as a per-user mailbox system without those extra pieces, the design becomes brittle quickly.

One-to-One Messaging Pattern

For direct messages, a common pattern is one shared topic for direct-message events, keyed by a stable conversation key or recipient key.

1import org.apache.kafka.clients.producer.KafkaProducer;
2import org.apache.kafka.clients.producer.ProducerRecord;
3import java.util.Properties;
4
5public class DirectMessageProducer {
6    public static void main(String[] args) {
7        Properties props = new Properties();
8        props.put("bootstrap.servers", "localhost:9092");
9        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
10        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
11
12        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
13            String conversationKey = "user-7:user-42";
14            String payload = "hello from user-7";
15            producer.send(new ProducerRecord<>("direct-messages", conversationKey, payload));
16            producer.flush();
17        }
18    }
19}

Keying by conversation or recipient matters because ordering is guaranteed only within a partition. If all events for one direct conversation use the same key, they stay in a stable order for consumers.

After consumption, a delivery service usually writes the event into a query-friendly store such as PostgreSQL, Cassandra, or Elasticsearch, depending on the product needs.

Group Messaging Pattern

Group messaging usually works better as a single group-message event followed by server-side fan-out. The producer does not send one Kafka record per member. It sends one event for the group, and a consumer resolves membership.

1import org.apache.kafka.clients.producer.KafkaProducer;
2import org.apache.kafka.clients.producer.ProducerRecord;
3import java.util.Properties;
4
5public class GroupMessageProducer {
6    public static void main(String[] args) {
7        Properties props = new Properties();
8        props.put("bootstrap.servers", "localhost:9092");
9        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
10        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
11
12        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
13            producer.send(new ProducerRecord<>("group-messages", "group-18", "standup moved to 10:30"));
14            producer.flush();
15        }
16    }
17}

A group-delivery consumer can then:

1. read the event from group-messages
2. look up current group membership
3. create one delivery record per member
4. emit secondary events for notifications, analytics, or search indexing

That keeps producers simple and centralizes membership rules in one place.

Consumer Design and Idempotency

Kafka consumers may retry work, rebalance, or reprocess records after failure. That means the message-handling code must be idempotent.

1import org.apache.kafka.clients.consumer.ConsumerRecord;
2
3public class MessageProcessor {
4    public void process(ConsumerRecord<String, String> record) {
5        String eventKey = record.key();
6        String payload = record.value();
7
8        // Persist using a unique message id or deduplication key.
9        System.out.println("processing " + eventKey + " -> " + payload);
10    }
11}

In a real system, the payload should contain a message identifier and schema version. Without a unique event id, retries can create duplicate inbox rows or duplicate push notifications.

Retention, Replay, and Delivery State

One reason Kafka is attractive for messaging backends is replay. If the search index breaks, or unread counts need rebuilding, retained events can rebuild derived state.

That benefit only helps if the event model is explicit. A useful messaging envelope normally includes:

• message id
• sender id
• conversation or group id
• creation timestamp
• payload schema version

Delivery and read state should not be inferred from Kafka offsets alone. Those are consumer mechanics, not user-visible message states. Track delivery events explicitly in downstream storage or in separate state topics.

Common Pitfalls

A common mistake is creating one topic per user or per conversation too early. Kafka can handle many topics, but operational overhead grows fast, and a single shared topic with stable keys is often the better design.

Another issue is forgetting the partition-key rule. If related messages are not keyed consistently, conversation order becomes unreliable.

Teams also often assume Kafka durability is the same thing as user delivery. It is not. A message can be safely stored in Kafka and still fail to reach the inbox, notification service, or mobile device.

Finally, consumer idempotency is not optional. In messaging systems, retries are normal, so duplicate downstream effects must be designed out deliberately.

Summary

• Kafka can support direct and group messaging, but the chat semantics are application-level design.
• Use stable partition keys to preserve per-conversation or per-group ordering.
• Direct messages usually map well to a shared topic plus downstream inbox storage.
• Group messages are usually best modeled as one event plus consumer-side fan-out.
• Delivery state, read state, and deduplication must be implemented explicitly outside raw Kafka transport.