Kubernetes
NATS
Debugging
Event Messaging
Cloud Infrastructure

Debugging NATS messages from Kubernetes event

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

When Kubernetes events are supposed to end up in NATS, there are three separate things to verify: the Kubernetes event source, the publisher that transforms or forwards the event, and the NATS subject where the message should appear. Debugging gets much easier once you stop treating it as one black box.

Start at the Kubernetes Side

First confirm the cluster is actually producing the event you expect.

bash
kubectl get events -A --sort-by=.lastTimestamp

If the event is not present in Kubernetes, there is nothing for the publisher to forward. Many NATS debugging sessions waste time at the messaging layer when the root cause is that the expected Kubernetes event never happened.

Find the Bridge Component

Usually a controller, sidecar, or event-exporter watches Kubernetes events and publishes them to NATS. Identify that component and inspect its logs.

bash
kubectl get pods -A | grep nats
kubectl logs -n my-namespace deployment/event-publisher

You are looking for evidence of three steps:

  • event received from the Kubernetes watch
  • event transformed into a message payload
  • publish attempt to a specific NATS subject

If the bridge never logs the event, the problem is upstream. If it logs a publish attempt but no consumer sees the message, the problem is downstream.

Subscribe Directly to the NATS Subject

The fastest way to debug the messaging layer is to subscribe directly.

bash
nats sub 'k8s.events.>'

If the NATS CLI is not available locally, run a temporary debug pod in the cluster or exec into a toolbox container that has NATS client utilities installed.

Seeing the raw subject and payload immediately answers a lot of questions:

  • is anything being published at all
  • is it going to the expected subject
  • is the payload shaped the way consumers expect

Verify the Subject Naming Convention

A very common problem is subject mismatch rather than message loss. The publisher may send to k8s.events.warning.default, while the subscriber listens only to k8s.events.default.

NATS is extremely fast, but it is also literal. If the subject names do not line up, nothing is delivered and there is no magic fallback.

That is why wildcard subscriptions such as k8s.events.> are so useful during debugging.

Inspect Payload Shape

Even when the message arrives, consumers can still reject it because the payload schema changed.

For example, an event exporter may publish JSON like this:

json
1{
2  "type": "Warning",
3  "reason": "BackOff",
4  "namespace": "default",
5  "message": "Back-off restarting failed container"
6}

If the consumer expects a different field set or nesting structure, the message can appear to vanish when in reality it was discarded by application logic after receipt.

Common Pitfalls

The biggest pitfall is starting with NATS before confirming the Kubernetes event exists. No event means no message.

Another issue is checking publisher logs without subscribing directly to the subject. Logs can say "published" while a subject mismatch or consumer bug still hides the real problem.

Developers also sometimes debug only one namespace or one subject pattern while the bridge publishes to another. Wildcard subscription is often the fastest diagnostic move.

Finally, remember that NATS debugging is not only transport debugging. Subject names and payload schema are part of the contract too.

Summary

  • Verify the Kubernetes event exists before debugging NATS.
  • Identify and inspect the bridge component that publishes cluster events into NATS.
  • Subscribe directly to the relevant NATS subject, ideally with a wildcard during debugging.
  • Check both subject naming and payload schema, not just transport-level delivery.
  • Treat the pipeline as three layers: source event, publisher bridge, and NATS subject consumer.

Course illustration
Course illustration

All Rights Reserved.