Is there a way to produce Kafka messages with headers using Kafka Confluent REST API?

Kafka

Confluent REST API

Message Headers

Data Production

Technology

Is there a way to produce Kafka messages with headers using Kafka Confluent REST API?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Many modern applications leverage Apache Kafka for event streaming, utilizing its robust architecture and scalability. In Kafka, messages can carry not only key-value pairs but also headers which can contain meta-information about the message. Often, users might wish to utilize Kafka’s Confluent REST Proxy for sending messages, which abstracts Kafka’s native protocol to HTTP, making it accessible for languages and frameworks not easily compatible with Kafka’s native clients. This article explores how to produce Kafka messages with headers using the Kafka Confluent REST API and assumes familiarity with basic Kafka concepts like topics, producers, and messages.

Overview: Kafka Confluent REST Proxy

The Confluent REST Proxy provides a RESTful interface to a Kafka cluster, making it simple to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka client. It is particularly useful for web applications and other environments where direct Kafka client integration is not optimal.

Producing Messages with Headers Using the Confluent REST API

Producing messages with headers through the Kafka Confluent REST API is straightforward. The REST API's /topics/(topic_name) endpoint allows for submitting messages. Here's how to do it in detail:

1. JSON Data Format

To send a message with headers, construct a JSON object containing the records you wish to send. Each record can contain multiple fields:

key: The message key (optional)
value: The message value
partition: The partition number (optional)
headers: A list of key-value pairs

2. HTTP Request

The JSON object is then sent as a POST request to the /topics/(topic_name) endpoint of your Kafka REST Proxy. The Content-Type should be set to application/vnd.kafka.json.v2+json.

Example Request

Here’s an example payload with headers. This payload is intended to be posted to the Kafka topic named test-topic:

json

1{
2  "records": [
3    {
4      "key": "key1",
5      "value": "value1",
6      "headers": [
7        {
8          "key": "headerKey1",
9          "value": "aGVhZGVyVmFsdWUx"  # base64 encoded
10        },
11        {
12          "key": "headerKey2",
13          "value": "aGVhZGVyVmFsdWUy"  # base64 encoded
14        }
15      ]
16    }
17  ]
18}

Note that headers must be base64 encoded. Various online tools or programming libraries can help you encode strings to base64.

CURL Request

Using CURL, the HTTP request would look like this:

bash

curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" \
     --data '{"records":[{"key":"key1","value":"value1","headers":[{"key":"headerKey1","value":"aGVhZGVyVmFsdWUx"},{"key":"headerKey2","value":"aGVhZGVyVmFsdWUy"}]}]}' \
     http://your-kafka-rest-proxy-host:8082/topics/test-topic

Table of HTTP Header Requirements

Here’s a quick summary of key data points and where to include them when producing messages:

Data Point	Requirement Format	Example Value
Content Type	Must be specific for JSON Encoding with Kafka	`application/vnd.kafka.json.v2+json`
Topic Name	Appended in the URL	`/topics/test-topic`
Message Content	JSON payload with `records` including keys and values	See JSON payload in the above example
Headers	Encoded as Base64 in `headers` array inside records	`"headers":[{"key":"example","value":"dGVzdA=="}]`

Summary

Producing messages with headers to Kafka via the Confluent REST Proxy is well-documented and supported, making it accessible for systems that prefer HTTP for messaging. Use cases include web services that interact with Kafka but are written in languages without robust Kafka client libraries or environments where rapid prototyping is required across different programming languages.

This integration flexibility opens Kafka's capabilities to a broader range of applications and should be considered a valuable tool for modern application architectures involving event-driven data streams. Ensure to handle base64 encoding correctly for headers to ensure data integrity and system interoperability.