Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. When you publish (or produce) a message to Kafka, you may choose how messages are sent: synchronously or asynchronously. This article focuses on synchronous message sending in Kafka, why it's important, how to implement it, and considerations for its use.

Understanding Synchronous Messaging in Kafka

In synchronous messaging, the producer sends a message to a Kafka broker and waits for an acknowledgement (ACK) from the broker before proceeding. This mode ensures reliability at the cost of latency and throughput. There are several configurations and considerations involved in implementing synchronous messaging in Kafka.

How to Implement Synchronous Messaging

To send messages synchronously, Kafka producers require specific configurations. Below is a step-by-step guide:

1. Configuration of the Producer:
   • acks: Set acks=all (or -1) which means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive.
   • retries: Set the number of retries in case of message send failure.
   • retry.backoff.ms: Set the time to wait before attempting to retry a failed send.

1Properties props = new Properties();
2props.put("bootstrap.servers", "localhost:9092");
3props.put("acks", "all");
4props.put("retries", 0);
5props.put("retry.backoff.ms", 100);
6props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
7props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
8
9Producer<String, String> producer = new KafkaProducer<>(props);

1. Sending a Message: Create a producer record and send it. Since Kafka is inherently asynchronous, to achieve synchronous behavior, the send() method is used along with get() to block the thread until it receives a response.

1try {
2    ProducerRecord<String, String> record = new ProducerRecord<>("TopicName", "key", "value");
3    producer.send(record).get(); // Synchronous send
4} catch (InterruptedException | ExecutionException e) {
5    e.printStackTrace();
6}

Trade-offs

While synchronous communication ensures reliable message delivery, it comes at the cost of performance. Here are few important trade-offs:

• Throughput vs. Durability: Synchronous sends are slower because the producer waits for acknowledgements, reducing throughput.
• Latency: Increased latencies as each message has to wait for the acknowledgment.
• Resource Utilization: Higher resource usage on the broker and the producer as more waiting and processing is involved.

Use Cases

Synchronous messaging is critical in scenarios where data loss cannot be tolerated, such as banking transactions or critical stateful operations. It is used less frequently in scenarios where latency and throughput are critical.

Summary Table

Advanced Considerations

• Network Configuration: Optimal network settings can help minimize the impact on latency and throughput.
• Batch Size: Configure producer batching along with synchronous sends to improve throughput.
• Timeouts: Appropriate timeout settings are necessary to prevent the system from hanging indefinitely in case of network or broker failures.

Conclusion

Implementing synchronous message delivery in Kafka provides guaranteed message delivery at the expense of latency and throughput. It's crucial to balance these factors based on the application's specific needs and the reliability required from your messaging system.