Monitoring HTTP Traffic in Kubernetes
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
Monitoring HTTP traffic in Kubernetes helps with debugging service communication, detecting performance bottlenecks, and identifying security issues. The main approaches are: using a service mesh (Istio, Linkerd) for automatic traffic metrics, deploying Prometheus + Grafana for metric collection and visualization, using kubectl port-forward with tools like tcpdump or curl for ad-hoc debugging, and injecting sidecar proxies for traffic inspection. The right approach depends on whether you need real-time debugging or long-term observability.
Method 1: kubectl for Quick Debugging
Port-Forward and Curl
Exec Into a Pod
tcpdump Inside a Pod
Method 2: Prometheus + Grafana
Deploy Prometheus
Annotate Services for Scraping
Useful Prometheus Queries
Method 3: Istio Service Mesh
Istio automatically injects sidecar proxies that capture all HTTP traffic:
Built-In Dashboards
Istio Traffic Metrics
Istio automatically generates metrics:
Method 4: Linkerd (Lightweight Service Mesh)
Linkerd output:
Method 5: Nginx Ingress Controller Metrics
Approach Comparison
| Approach | Setup | Overhead | Real-Time | Historical | Best For |
| kubectl + curl | None | None | Yes | No | Quick debugging |
| Prometheus + Grafana | Medium | Low | Yes | Yes | Application metrics |
| Istio | High | Medium | Yes | Yes | Full mesh observability |
| Linkerd | Medium | Low | Yes | Yes | Lightweight mesh |
| tcpdump/netshoot | None | None | Yes | No | Packet-level debugging |
Common Pitfalls
- Not exposing a
/metricsendpoint from your application: Prometheus can only scrape metrics that your application exposes. Use a metrics library (Prometheus client for Go/Python/Java/Node) to exposehttp_requests_total,http_request_duration_seconds, etc. Without application-level metrics, you only see infrastructure-level data. - Injecting a service mesh into production without testing: Service mesh sidecars (Istio, Linkerd) add latency (1-5ms per hop) and memory overhead (~50-100MB per pod). Test in staging first and measure the performance impact before enabling in production.
- Capturing too much traffic with tcpdump: Running
tcpdumpwithout filters on a busy pod captures everything, producing massive output and potentially impacting pod performance. Always filter by port (port 80), host, or protocol. - Using port-forward for load testing:
kubectl port-forwardtunnels through the API server and is not designed for high throughput. Use it only for debugging single requests, not performance testing. - Forgetting that Kubernetes DNS changes after pod restarts: Service-to-service communication uses DNS names like
my-service.namespace.svc.cluster.local. If pods restart and DNS caching is aggressive, old connections may fail. Monitor DNS resolution alongside HTTP traffic.
Summary
- Use
kubectl exec+curlfor quick ad-hoc debugging of inter-service communication - Deploy Prometheus + Grafana for long-term HTTP metrics collection and alerting
- Use Istio or Linkerd for automatic traffic monitoring across all services in the mesh
- Expose
/metricsendpoints from your applications for Prometheus to scrape - Use ephemeral debug containers (
kubectl debug) for packet-level inspection with tcpdump
Related reading
- Mount add files to existing directory using configmap volume mount
- Mount local directory into pod in minikube
- Mounting kubernetes volume with User permission
- Mounting NFS Persistent Volumes with authentication
- Monitoring pending async operations in Node.js promised environment
- Monitoring UI for Apache kafka - kafka manager vs kafka monitor
- More classes in config than trained on tensorflow object detection API
- MountVolume.SetUp failed for volume kube-api-access-fcz9j object default/kube-root-ca.crt not registered

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.