Introduction

Sending Filebeat events to multiple Kafka topics is a common way to separate application logs, security logs, and environment-specific traffic. In Filebeat 5.0-era configurations, the usual pattern is to assign a routing field to each event and use that field when building the Kafka topic name.

Start With a Stable Kafka Output

Before adding dynamic topic routing, define the Kafka output with the reliability and size settings your brokers expect.

1output.kafka:
2  hosts: ["kafka1:9092", "kafka2:9092"]
3  required_acks: 1
4  compression: gzip
5  max_message_bytes: 1000000

These settings should match broker-side limits. Topic routing is much easier to debug when the baseline producer configuration is already known to work.

Put the Routing Hint on the Event

In Filebeat 5.0, a common approach is to attach a custom field on each prospector so the event carries its intended topic.

1filebeat.prospectors:
2  - input_type: log
3    paths:
4      - /var/log/app/*.log
5    fields:
6      log_topic: app-logs
7    fields_under_root: true
8
9  - input_type: log
10    paths:
11      - /var/log/security/*.log
12    fields:
13      log_topic: security-logs
14    fields_under_root: true

This makes the routing rule explicit and easy to audit. Each input path declares where its events should land.

Build the Kafka Topic Dynamically

Once the event contains the routing field, the Kafka output can reference it in the topic setting.

output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092"]
  topic: '%{[log_topic]}'

That is enough for simple multi-topic routing. It keeps Filebeat configuration readable because the topic decision is visible in one place and the output stanza does not need a separate hard-coded block per topic.

Add Environment Prefixes Carefully

Many teams want topic names such as prod-app-logs and prod-security-logs. That is fine, but keep the topic name construction bounded and predictable.

1fields:
2  env: prod
3
4output.kafka:
5  hosts: ["kafka1:9092"]
6  topic: '%{[env]}-%{[log_topic]}'

The important part is control. Do not let arbitrary log fields generate topic names or you risk topic sprawl, broken consumers, and difficult operations. Topic naming should be governed, not emergent.

Define a Fallback Strategy

A dynamic topic template is only safe if you know what happens when the routing field is missing. Depending on your broader pipeline, that could mean rejecting the event, sending it to a default topic, or fixing the input configuration.

A practical approach is to test every branch:

1. expected app log event
2. expected security event
3. malformed event with no routing field

The goal is to make routing failure obvious instead of silently dropping or misclassifying data.

Validate End to End With Kafka Consumers

Do not stop at Filebeat startup success. Confirm that real events arrive in the intended topics with consumer checks.

1kafka-console-consumer \
2  --bootstrap-server kafka1:9092 \
3  --topic prod-app-logs \
4  --from-beginning
5
6kafka-console-consumer \
7  --bootstrap-server kafka1:9092 \
8  --topic prod-security-logs \
9  --from-beginning

This catches both routing mistakes and broker-side problems. It is especially important when you are migrating an older Filebeat configuration or changing topic naming conventions.

Keep Operations in Mind

Multi-topic output is not just a configuration trick. It creates operational requirements around topic ownership, retention policies, ACLs, and monitoring. Without governance, a clever dynamic-topic setup becomes an expensive support burden.

At minimum, define:

1. approved topic names
2. who owns each topic
3. expected retention class
4. alerts for unknown or missing topics

That structure keeps the routing logic useful instead of chaotic.

Common Pitfalls

The biggest mistake is routing based on a field that is not present on every event path. Another is constructing topic names from uncontrolled values, which creates a topic explosion problem. Teams also validate only one happy path and miss the fact that another prospector is silently publishing to the wrong topic or no topic at all.

Summary

• Multi-topic Kafka output in Filebeat 5.0 is usually driven by event fields plus a dynamic topic template.
• Set a stable Kafka output baseline before adding routing logic.
• Attach topic hints explicitly at the prospector level.
• Keep topic naming bounded and validate every routing branch end to end.
• Treat topic governance as part of the design, not as cleanup after deployment.