Introduction
Monitoring HTTP traffic in Kubernetes helps with debugging service communication, detecting performance bottlenecks, and identifying security issues. The main approaches are: using a service mesh (Istio, Linkerd) for automatic traffic metrics, deploying Prometheus + Grafana for metric collection and visualization, using kubectl port-forward with tools like tcpdump or curl for ad-hoc debugging, and injecting sidecar proxies for traffic inspection. The right approach depends on whether you need real-time debugging or long-term observability.
Method 1: kubectl for Quick Debugging
Port-Forward and Curl
1# Forward a service port to your local machine
2kubectl port-forward svc/my-api-service 8080:80
3
4# Test HTTP endpoints
5curl http://localhost:8080/health
6curl -v http://localhost:8080/api/users # Verbose output with headers
Exec Into a Pod
1# Run curl from inside a pod
2kubectl exec -it my-pod -- curl http://other-service.namespace.svc.cluster.local/api/data
3
4# Install curl if not available
5kubectl exec -it my-pod -- sh -c "apt-get update && apt-get install -y curl"
6
7# Use wget (often available in Alpine-based images)
8kubectl exec -it my-pod -- wget -qO- http://other-service/health
tcpdump Inside a Pod
1# Capture traffic on a pod's network interface
2kubectl exec -it my-pod -- tcpdump -i any -A port 80
3
4# Filter for specific HTTP traffic
5kubectl exec -it my-pod -- tcpdump -i any -s 0 -A 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
6
7# If tcpdump is not installed, use an ephemeral debug container
8kubectl debug -it my-pod --image=nicolaka/netshoot --target=my-container
9tcpdump -i any port 80
Method 2: Prometheus + Grafana
Deploy Prometheus
1# prometheus-config.yaml
2apiVersion: v1
3kind: ConfigMap
4metadata:
5 name: prometheus-config
6data:
7 prometheus.yml: |
8 global:
9 scrape_interval: 15s
10 scrape_configs:
11 - job_name: 'kubernetes-services'
12 kubernetes_sd_configs:
13 - role: endpoints
14 relabel_configs:
15 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
16 action: keep
17 regex: true
18 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
19 action: replace
20 target_label: __address__
21 regex: (.+)
Annotate Services for Scraping
1apiVersion: v1
2kind: Service
3metadata:
4 name: my-api
5 annotations:
6 prometheus.io/scrape: "true"
7 prometheus.io/port: "8080"
8 prometheus.io/path: "/metrics"
9spec:
10 selector:
11 app: my-api
12 ports:
13 - port: 80
14 targetPort: 8080
Useful Prometheus Queries
1# Request rate per service
2rate(http_requests_total{job="my-api"}[5m])
3
4# 95th percentile response time
5histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="my-api"}[5m]))
6
7# Error rate (5xx responses)
8sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
9
10# Requests per endpoint
11sum by (path, method) (rate(http_requests_total{job="my-api"}[5m]))
Method 3: Istio Service Mesh
Istio automatically injects sidecar proxies that capture all HTTP traffic:
1# Install Istio
2istioctl install --set profile=demo
3
4# Enable injection for a namespace
5kubectl label namespace default istio-injection=enabled
6
7# Restart pods to inject sidecars
8kubectl rollout restart deployment my-api
Built-In Dashboards
1# Kiali — service mesh visualization
2kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
3istioctl dashboard kiali
4
5# Jaeger — distributed tracing
6kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
7istioctl dashboard jaeger
8
9# Grafana — metrics dashboards
10kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml
11istioctl dashboard grafana
Istio Traffic Metrics
Istio automatically generates metrics:
1# Request volume
2istio_requests_total{destination_service="my-api.default.svc.cluster.local"}
3
4# Response latency
5istio_request_duration_milliseconds_bucket{destination_service="my-api.default.svc.cluster.local"}
6
7# Error rate by source
8sum(rate(istio_requests_total{response_code=~"5.*", destination_service="my-api"}[5m]))
9 by (source_workload)
Method 4: Linkerd (Lightweight Service Mesh)
1# Install Linkerd
2linkerd install | kubectl apply -f -
3
4# Inject sidecar into a deployment
5kubectl get deployment my-api -o yaml | linkerd inject - | kubectl apply -f -
6
7# View live traffic
8linkerd viz stat deployments
9linkerd viz top deployment/my-api
10
11# View per-route metrics
12linkerd viz routes deployment/my-api
Linkerd output:
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
my-api 1/1 100.00% 45 2ms 10ms 25ms
Method 5: Nginx Ingress Controller Metrics
1# Enable metrics in the Ingress controller
2apiVersion: v1
3kind: ConfigMap
4metadata:
5 name: nginx-configuration
6data:
7 enable-vts-status: "true"
1# Ingress request rate
2sum(rate(nginx_ingress_controller_requests[5m])) by (ingress, status)
3
4# Response time per ingress
5histogram_quantile(0.95,
6 sum(rate(nginx_ingress_controller_request_duration_seconds_bucket[5m])) by (le, ingress))
Approach Comparison
| Approach | Setup | Overhead | Real-Time | Historical | Best For |
| kubectl + curl | None | None | Yes | No | Quick debugging |
| Prometheus + Grafana | Medium | Low | Yes | Yes | Application metrics |
| Istio | High | Medium | Yes | Yes | Full mesh observability |
| Linkerd | Medium | Low | Yes | Yes | Lightweight mesh |
| tcpdump/netshoot | None | None | Yes | No | Packet-level debugging |
Common Pitfalls
Not exposing a /metrics endpoint from your application: Prometheus can only scrape metrics that your application exposes. Use a metrics library (Prometheus client for Go/Python/Java/Node) to expose http_requests_total, http_request_duration_seconds, etc. Without application-level metrics, you only see infrastructure-level data.
Injecting a service mesh into production without testing: Service mesh sidecars (Istio, Linkerd) add latency (1-5ms per hop) and memory overhead (~50-100MB per pod). Test in staging first and measure the performance impact before enabling in production.
Capturing too much traffic with tcpdump: Running tcpdump without filters on a busy pod captures everything, producing massive output and potentially impacting pod performance. Always filter by port (port 80), host, or protocol.
Using port-forward for load testing: kubectl port-forward tunnels through the API server and is not designed for high throughput. Use it only for debugging single requests, not performance testing.
Forgetting that Kubernetes DNS changes after pod restarts: Service-to-service communication uses DNS names like my-service.namespace.svc.cluster.local. If pods restart and DNS caching is aggressive, old connections may fail. Monitor DNS resolution alongside HTTP traffic.
Summary
Use kubectl exec + curl for quick ad-hoc debugging of inter-service communication
Deploy Prometheus + Grafana for long-term HTTP metrics collection and alerting
Use Istio or Linkerd for automatic traffic monitoring across all services in the mesh
Expose /metrics endpoints from your applications for Prometheus to scrape
Use ephemeral debug containers (kubectl debug) for packet-level inspection with tcpdump
