Kafka
API
Unit Testing
Mocking
Software Development

Mocking Kafka APIs for Unit Testing

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Apache Kafka is a widely used distributed event streaming platform that handles trillions of events every day. However, testing applications that use Kafka can often be challenging due to its scale, cluster dependency, and asynchronous nature. Mocking Kafka APIs for unit testing becomes essential to ensure the components interact with Kafka as expected without the overhead of setting up an actual Kafka cluster. This article explains how to effectively mock Kafka APIs using different tools and frameworks.

Why Mock Kafka?

Mocking Kafka is crucial for several reasons:

  • Speed: Tests run significantly faster as they do not need to interact with an actual Kafka broker.
  • Isolation: It helps in testing the system in isolation, preventing tests from failing due to unrelated issues like network failures.
  • Control: Allows control over Kafka’s behavior in test scenarios, making it possible to test all possible edge cases.
  • Resources: Reduces the resources required for testing because you do not need to maintain a Kafka environment for testing.

Tools and Libraries for Mocking Kafka

Several tools and libraries can be used to mock Kafka APIs:

  • Mockito – Useful for mocking Kafka producer and consumer APIs at the class or interface level.
  • Embedded Kafka – A real Kafka server runs in a sandbox environment ideal for integration tests rather than unit tests.
  • Testcontainers – Uses Docker to spin up Kafka containers, providing a more realistic test environment.
  • Spring Kafka Test – Provides @EmbeddedKafka annotation to integrate an embedded Kafka broker when using Spring Boot.

1. Mocking Kafka Producer

When unit testing classes that produce messages to Kafka, you need to ensure that these messages are produced as expected based on various inputs and conditions.

Here's an example of how to mock Kafka's Producer using Mockito:

java
1import org.apache.kafka.clients.producer.Producer;
2import org.apache.kafka.clients.producer.ProducerRecord;
3import org.apache.kafka.clients.producer.RecordMetadata;
4import org.mockito.Mockito;
5
6public class KafkaProducerTest {
7    private Producer<String, String> producer;
8
9    @BeforeEach
10    void setUp() {
11        producer = Mockito.mock(Producer.class);
12    }
13
14    @Test
15    void testSendMessage() {
16        String topic = "test-topic";
17        String key = "key1";
18        String value = "value1";
19
20        // Given
21        Mockito.doAnswer(invocation -> {
22            ProducerRecord<String, String> record = invocation.getArgument(0);
23            assertEquals(topic, record.topic());
24            assertEquals(key, record.key());
25            assertEquals(value, record.value());
26            return new RecordMetadata(new TopicPartition(topic, 1), 0, 0, System.currentTimeMillis(), Long.valueOf(-1), -1, -1);
27        }).when(producer).send(Mockito.any(ProducerRecord.class));
28
29        // When
30        producer.send(new ProducerRecord<>(topic, key, value));
31
32        // Then
33        Mockito.verify(producer).send(any(ProducerRecord.class));
34    }
35}

2. Mocking Kafka Consumer

Testing Kafka consumers involves ensuring that the consumer processes messages correctly when they are read from a Kafka topic.

Example of how to mock Kafka's Consumer using Mockito:

java
1import org.apache.kafka.clients.consumer.Consumer;
2import org.apache.kafka.clients.consumer.ConsumerRecord;
3import org.apache.kafka.clients.consumer.ConsumerRecords;
4import org.mockito.Mockito;
5
6public class KafkaConsumerTest {
7    private Consumer<String, String> consumer;
8
9    @BeforeEach
10    void setUp() {
11        consumer = Mockito.mock(Consumer.class);
12    }
13
14    @Test
15    void testConsumeMessage() {
16        String topic = "test-topic";
17        String key = "key1";
18        String value = "value1";
19
20        // Given
21        HashMap<TopicPartition, List<ConsumerRecord<String, String>>> records = new HashMap<>();
22        records.put(new TopicPartition(topic, 0), Arrays.asList(new ConsumerRecord<>(topic, 0, 0L, key, value)));
23        ConsumerRecords<String, String> consumerRecords = new ConsumerRecords<>(records);
24
25        Mockito.when(consumer.poll(any(Duration.class))).thenReturn(consumerRecords);
26
27        // When
28        ConsumerRecords<String, String> result = consumer.poll(Duration.ofMillis(100));
29
30        // Then
31        assertEquals(1, result.count());
32        ConsumerRecord<String, String> record = result.iterator().next();
33        assertEquals(topic, record.topic());
34        assertEquals(key, record.key());
35        assertEquals(value, record.value());
36    }
37}

Summary Table

AspectTool/LibraryUse CaseImplementation Level
MockingMockitoUnit testing Kafka producers/consumersClass/Interface level
Embedded KafkaSpring Kafka TestIntegration testing with an actual brokerApplication level
ContainerizedTestcontainersIntegration testing in isolated containersSystem level

Conclusion

Mocking Kafka APIs is a fundamental aspect of developing robust Kafka-based applications. Tools like Mockito and frameworks like Spring Kafka Test offer powerful options to ensure your Kafka interactions are thoroughly tested without requiring a full Kafka setup. This leads to faster, more reliable testing cycles. By choosing the appropriate mocking technique and library, developers can simulate most aspects of Kafka behavior and focus on the correctness and efficiency of business logic.


Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.