Debugging NATS messages from Kubernetes event
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
When Kubernetes events are supposed to end up in NATS, there are three separate things to verify: the Kubernetes event source, the publisher that transforms or forwards the event, and the NATS subject where the message should appear. Debugging gets much easier once you stop treating it as one black box.
Start at the Kubernetes Side
First confirm the cluster is actually producing the event you expect.
If the event is not present in Kubernetes, there is nothing for the publisher to forward. Many NATS debugging sessions waste time at the messaging layer when the root cause is that the expected Kubernetes event never happened.
Find the Bridge Component
Usually a controller, sidecar, or event-exporter watches Kubernetes events and publishes them to NATS. Identify that component and inspect its logs.
You are looking for evidence of three steps:
- event received from the Kubernetes watch
- event transformed into a message payload
- publish attempt to a specific NATS subject
If the bridge never logs the event, the problem is upstream. If it logs a publish attempt but no consumer sees the message, the problem is downstream.
Subscribe Directly to the NATS Subject
The fastest way to debug the messaging layer is to subscribe directly.
If the NATS CLI is not available locally, run a temporary debug pod in the cluster or exec into a toolbox container that has NATS client utilities installed.
Seeing the raw subject and payload immediately answers a lot of questions:
- is anything being published at all
- is it going to the expected subject
- is the payload shaped the way consumers expect
Verify the Subject Naming Convention
A very common problem is subject mismatch rather than message loss. The publisher may send to k8s.events.warning.default, while the subscriber listens only to k8s.events.default.
NATS is extremely fast, but it is also literal. If the subject names do not line up, nothing is delivered and there is no magic fallback.
That is why wildcard subscriptions such as k8s.events.> are so useful during debugging.
Inspect Payload Shape
Even when the message arrives, consumers can still reject it because the payload schema changed.
For example, an event exporter may publish JSON like this:
If the consumer expects a different field set or nesting structure, the message can appear to vanish when in reality it was discarded by application logic after receipt.
Common Pitfalls
The biggest pitfall is starting with NATS before confirming the Kubernetes event exists. No event means no message.
Another issue is checking publisher logs without subscribing directly to the subject. Logs can say "published" while a subject mismatch or consumer bug still hides the real problem.
Developers also sometimes debug only one namespace or one subject pattern while the bridge publishes to another. Wildcard subscription is often the fastest diagnostic move.
Finally, remember that NATS debugging is not only transport debugging. Subject names and payload schema are part of the contract too.
Summary
- Verify the Kubernetes event exists before debugging NATS.
- Identify and inspect the bridge component that publishes cluster events into NATS.
- Subscribe directly to the relevant NATS subject, ideally with a wildcard during debugging.
- Check both subject naming and payload schema, not just transport-level delivery.
- Treat the pipeline as three layers: source event, publisher bridge, and NATS subject consumer.

