When setting up an in-memory or embedded Kafka instance, it is common to think of it as a tool exclusively for testing purposes. However, there are scenarios outside of testing where such a setup can be incredibly useful, like in development environments, during prototyping, or for educational purposes. This article will guide you through implementing an in-memory or embedded Kafka, focusing primarily on non-testing applications.

Understanding Embedded and In-Memory Kafka

Embedded Kafka refers to running Kafka embedded within another Java application, typically for testing purposes. The Kafka server runs in the same JVM as the application, easing configuration, management, and interaction. In-Memory Kafka, on the other hand, is typically a subset or an extended version of embedded Kafka and does not persist data to disk, making it volatile and fast.

Why Use it Outside Testing?

• Development: Facilitates developers to work with Kafka without needing to set up an external cluster.
• Prototyping: Quickly prototype ideas without setting up complex infrastructure.
• Training and Education: Allows learners to experiment with Kafka without any setup overhead.

Key Components

• Apache Kafka: The core brokers that handle message storage and distribution.
• ZooKeeper: Manages the Kafka cluster configuration and coordination.
• Kafka Streams: For stream processing directly integrated into the JVM application.

Setup Using Kafka Streams

Kafka Streams includes an in-memory state store and stream processing capabilities, which can provide a light impression of using Kafka in-memory. Below is a simple setup:

1Properties config = new Properties();
2config.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-application");
3config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
4config.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
5config.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
6
7StreamsBuilder builder = new StreamsBuilder();
8builder.<String, String>stream("input-topic")
9       .filter((key, value) -> value.length() > 5)
10       .to("output-topic");
11
12KafkaStreams streams = new KafkaStreams(builder.build(), config);
13streams.start();

This simple Kafka Streams application reads from the input-topic, filters messages based on value length, and writes to the output-topic. Kafka Streams handles the heavy lifting of interacting with the Kafka broker.

Embedded Kafka with Spring

For a more genuine in-memory or embedded usage, you can integrate Kafka with Spring Kafka:

1@EmbeddedKafka(partitions = 1, topics = { "topic1" })
2@SpringBootTest(classes = KafkaConsumerApplication.class)
3public class KafkaEmbeddedTest {
4
5    @Autowired
6    private KafkaConsumer consumer;
7
8    @Test
9    public void testReceivingKafkaEvents() {
10        kafkaTemplate.send("topic1", "Hello World");
11        ConsumerRecord<String, String> received = getSingleRecord(consumer, "topic1");
12        assertThat(received.value()).isEqualTo("Hello World");
13    }
14}

This example initializes an embedded Kafka broker and interacts with it through a Spring-based Kafka consumer. Even though this is set up in a test (annotated with @Test), the same approach can be applied outside tests by configuring your Spring application correctly.

Considerations

Conclusion

Using in-memory or embedded Kafka outside of testing environments can be beneficial for development, learning, and experimentation. However, understanding its limitations in terms of data persistence, resource usage, and scalability is crucial. Always ensure the Kafka setup aligns with the application's goals and requirements. For actual production environments, a dedicated Kafka cluster is recommended.