Mocking Kafka APIs for Unit Testing

Kafka

API

Unit Testing

Mocking

Software Development

Mocking Kafka APIs for Unit Testing

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Kafka & Streaming Networking & APIs Testing

Apache Kafka is a widely used distributed event streaming platform that handles trillions of events every day. However, testing applications that use Kafka can often be challenging due to its scale, cluster dependency, and asynchronous nature. Mocking Kafka APIs for unit testing becomes essential to ensure the components interact with Kafka as expected without the overhead of setting up an actual Kafka cluster. This article explains how to effectively mock Kafka APIs using different tools and frameworks.

Why Mock Kafka?

Mocking Kafka is crucial for several reasons:

Speed: Tests run significantly faster as they do not need to interact with an actual Kafka broker.
Isolation: It helps in testing the system in isolation, preventing tests from failing due to unrelated issues like network failures.
Control: Allows control over Kafka’s behavior in test scenarios, making it possible to test all possible edge cases.
Resources: Reduces the resources required for testing because you do not need to maintain a Kafka environment for testing.

Tools and Libraries for Mocking Kafka

Several tools and libraries can be used to mock Kafka APIs:

Mockito – Useful for mocking Kafka producer and consumer APIs at the class or interface level.
Embedded Kafka – A real Kafka server runs in a sandbox environment ideal for integration tests rather than unit tests.
Testcontainers – Uses Docker to spin up Kafka containers, providing a more realistic test environment.
Spring Kafka Test – Provides @EmbeddedKafka annotation to integrate an embedded Kafka broker when using Spring Boot.

1. Mocking Kafka Producer

When unit testing classes that produce messages to Kafka, you need to ensure that these messages are produced as expected based on various inputs and conditions.

Here's an example of how to mock Kafka's Producer using Mockito:

java

1import org.apache.kafka.clients.producer.Producer;
2import org.apache.kafka.clients.producer.ProducerRecord;
3import org.apache.kafka.clients.producer.RecordMetadata;
4import org.mockito.Mockito;
5
6public class KafkaProducerTest {
7    private Producer<String, String> producer;
8
9    @BeforeEach
10    void setUp() {
11        producer = Mockito.mock(Producer.class);
12    }
13
14    @Test
15    void testSendMessage() {
16        String topic = "test-topic";
17        String key = "key1";
18        String value = "value1";
19
20        // Given
21        Mockito.doAnswer(invocation -> {
22            ProducerRecord<String, String> record = invocation.getArgument(0);
23            assertEquals(topic, record.topic());
24            assertEquals(key, record.key());
25            assertEquals(value, record.value());
26            return new RecordMetadata(new TopicPartition(topic, 1), 0, 0, System.currentTimeMillis(), Long.valueOf(-1), -1, -1);
27        }).when(producer).send(Mockito.any(ProducerRecord.class));
28
29        // When
30        producer.send(new ProducerRecord<>(topic, key, value));
31
32        // Then
33        Mockito.verify(producer).send(any(ProducerRecord.class));
34    }
35}

2. Mocking Kafka Consumer

Testing Kafka consumers involves ensuring that the consumer processes messages correctly when they are read from a Kafka topic.

Example of how to mock Kafka's Consumer using Mockito:

java

1import org.apache.kafka.clients.consumer.Consumer;
2import org.apache.kafka.clients.consumer.ConsumerRecord;
3import org.apache.kafka.clients.consumer.ConsumerRecords;
4import org.mockito.Mockito;
5
6public class KafkaConsumerTest {
7    private Consumer<String, String> consumer;
8
9    @BeforeEach
10    void setUp() {
11        consumer = Mockito.mock(Consumer.class);
12    }
13
14    @Test
15    void testConsumeMessage() {
16        String topic = "test-topic";
17        String key = "key1";
18        String value = "value1";
19
20        // Given
21        HashMap<TopicPartition, List<ConsumerRecord<String, String>>> records = new HashMap<>();
22        records.put(new TopicPartition(topic, 0), Arrays.asList(new ConsumerRecord<>(topic, 0, 0L, key, value)));
23        ConsumerRecords<String, String> consumerRecords = new ConsumerRecords<>(records);
24
25        Mockito.when(consumer.poll(any(Duration.class))).thenReturn(consumerRecords);
26
27        // When
28        ConsumerRecords<String, String> result = consumer.poll(Duration.ofMillis(100));
29
30        // Then
31        assertEquals(1, result.count());
32        ConsumerRecord<String, String> record = result.iterator().next();
33        assertEquals(topic, record.topic());
34        assertEquals(key, record.key());
35        assertEquals(value, record.value());
36    }
37}

Summary Table

Aspect	Tool/Library	Use Case	Implementation Level
Mocking	Mockito	Unit testing Kafka producers/consumers	Class/Interface level
Embedded Kafka	Spring Kafka Test	Integration testing with an actual broker	Application level
Containerized	Testcontainers	Integration testing in isolated containers	System level

Conclusion

Mocking Kafka APIs is a fundamental aspect of developing robust Kafka-based applications. Tools like Mockito and frameworks like Spring Kafka Test offer powerful options to ensure your Kafka interactions are thoroughly tested without requiring a full Kafka setup. This leads to faster, more reliable testing cycles. By choosing the appropriate mocking technique and library, developers can simulate most aspects of Kafka behavior and focus on the correctness and efficiency of business logic.

System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Mocking Kafka APIs for Unit Testing

System Design practice on Codemia

Why Mock Kafka?

Tools and Libraries for Mocking Kafka

1. Mocking Kafka Producer

2. Mocking Kafka Consumer

Summary Table

Conclusion

Related reading

System Design Fundamentals

Track what you have practised

System Design practice on Codemia