Practice Now
Kubernetes
HTTP Traffic
Monitoring
Network Monitoring
Containerization

Monitoring HTTP Traffic in Kubernetes

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
KubernetesDevOps & DeploymentNetworking & APIs

Introduction

Monitoring HTTP traffic in Kubernetes helps with debugging service communication, detecting performance bottlenecks, and identifying security issues. The main approaches are: using a service mesh (Istio, Linkerd) for automatic traffic metrics, deploying Prometheus + Grafana for metric collection and visualization, using kubectl port-forward with tools like tcpdump or curl for ad-hoc debugging, and injecting sidecar proxies for traffic inspection. The right approach depends on whether you need real-time debugging or long-term observability.

Method 1: kubectl for Quick Debugging

Port-Forward and Curl

bash
1# Forward a service port to your local machine
2kubectl port-forward svc/my-api-service 8080:80
3
4# Test HTTP endpoints
5curl http://localhost:8080/health
6curl -v http://localhost:8080/api/users  # Verbose output with headers

Exec Into a Pod

bash
1# Run curl from inside a pod
2kubectl exec -it my-pod -- curl http://other-service.namespace.svc.cluster.local/api/data
3
4# Install curl if not available
5kubectl exec -it my-pod -- sh -c "apt-get update && apt-get install -y curl"
6
7# Use wget (often available in Alpine-based images)
8kubectl exec -it my-pod -- wget -qO- http://other-service/health

tcpdump Inside a Pod

bash
1# Capture traffic on a pod's network interface
2kubectl exec -it my-pod -- tcpdump -i any -A port 80
3
4# Filter for specific HTTP traffic
5kubectl exec -it my-pod -- tcpdump -i any -s 0 -A 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
6
7# If tcpdump is not installed, use an ephemeral debug container
8kubectl debug -it my-pod --image=nicolaka/netshoot --target=my-container
9tcpdump -i any port 80

Method 2: Prometheus + Grafana

Deploy Prometheus

yaml
1# prometheus-config.yaml
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: prometheus-config
6data:
7  prometheus.yml: |
8    global:
9      scrape_interval: 15s
10    scrape_configs:
11      - job_name: 'kubernetes-services'
12        kubernetes_sd_configs:
13          - role: endpoints
14        relabel_configs:
15          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
16            action: keep
17            regex: true
18          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
19            action: replace
20            target_label: __address__
21            regex: (.+)

Annotate Services for Scraping

yaml
1apiVersion: v1
2kind: Service
3metadata:
4  name: my-api
5  annotations:
6    prometheus.io/scrape: "true"
7    prometheus.io/port: "8080"
8    prometheus.io/path: "/metrics"
9spec:
10  selector:
11    app: my-api
12  ports:
13    - port: 80
14      targetPort: 8080

Useful Prometheus Queries

promql
1# Request rate per service
2rate(http_requests_total{job="my-api"}[5m])
3
4# 95th percentile response time
5histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="my-api"}[5m]))
6
7# Error rate (5xx responses)
8sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
9
10# Requests per endpoint
11sum by (path, method) (rate(http_requests_total{job="my-api"}[5m]))

Method 3: Istio Service Mesh

Istio automatically injects sidecar proxies that capture all HTTP traffic:

bash
1# Install Istio
2istioctl install --set profile=demo
3
4# Enable injection for a namespace
5kubectl label namespace default istio-injection=enabled
6
7# Restart pods to inject sidecars
8kubectl rollout restart deployment my-api

Built-In Dashboards

bash
1# Kiali — service mesh visualization
2kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
3istioctl dashboard kiali
4
5# Jaeger — distributed tracing
6kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
7istioctl dashboard jaeger
8
9# Grafana — metrics dashboards
10kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml
11istioctl dashboard grafana

Istio Traffic Metrics

Istio automatically generates metrics:

promql
1# Request volume
2istio_requests_total{destination_service="my-api.default.svc.cluster.local"}
3
4# Response latency
5istio_request_duration_milliseconds_bucket{destination_service="my-api.default.svc.cluster.local"}
6
7# Error rate by source
8sum(rate(istio_requests_total{response_code=~"5.*", destination_service="my-api"}[5m]))
9  by (source_workload)

Method 4: Linkerd (Lightweight Service Mesh)

bash
1# Install Linkerd
2linkerd install | kubectl apply -f -
3
4# Inject sidecar into a deployment
5kubectl get deployment my-api -o yaml | linkerd inject - | kubectl apply -f -
6
7# View live traffic
8linkerd viz stat deployments
9linkerd viz top deployment/my-api
10
11# View per-route metrics
12linkerd viz routes deployment/my-api

Linkerd output:

 
NAME     MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
my-api   1/1      100.00%   45    2ms           10ms          25ms

Method 5: Nginx Ingress Controller Metrics

yaml
1# Enable metrics in the Ingress controller
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: nginx-configuration
6data:
7  enable-vts-status: "true"
promql
1# Ingress request rate
2sum(rate(nginx_ingress_controller_requests[5m])) by (ingress, status)
3
4# Response time per ingress
5histogram_quantile(0.95,
6  sum(rate(nginx_ingress_controller_request_duration_seconds_bucket[5m])) by (le, ingress))

Approach Comparison

ApproachSetupOverheadReal-TimeHistoricalBest For
kubectl + curlNoneNoneYesNoQuick debugging
Prometheus + GrafanaMediumLowYesYesApplication metrics
IstioHighMediumYesYesFull mesh observability
LinkerdMediumLowYesYesLightweight mesh
tcpdump/netshootNoneNoneYesNoPacket-level debugging

Common Pitfalls

  • Not exposing a /metrics endpoint from your application: Prometheus can only scrape metrics that your application exposes. Use a metrics library (Prometheus client for Go/Python/Java/Node) to expose http_requests_total, http_request_duration_seconds, etc. Without application-level metrics, you only see infrastructure-level data.
  • Injecting a service mesh into production without testing: Service mesh sidecars (Istio, Linkerd) add latency (1-5ms per hop) and memory overhead (~50-100MB per pod). Test in staging first and measure the performance impact before enabling in production.
  • Capturing too much traffic with tcpdump: Running tcpdump without filters on a busy pod captures everything, producing massive output and potentially impacting pod performance. Always filter by port (port 80), host, or protocol.
  • Using port-forward for load testing: kubectl port-forward tunnels through the API server and is not designed for high throughput. Use it only for debugging single requests, not performance testing.
  • Forgetting that Kubernetes DNS changes after pod restarts: Service-to-service communication uses DNS names like my-service.namespace.svc.cluster.local. If pods restart and DNS caching is aggressive, old connections may fail. Monitor DNS resolution alongside HTTP traffic.

Summary

  • Use kubectl exec + curl for quick ad-hoc debugging of inter-service communication
  • Deploy Prometheus + Grafana for long-term HTTP metrics collection and alerting
  • Use Istio or Linkerd for automatic traffic monitoring across all services in the mesh
  • Expose /metrics endpoints from your applications for Prometheus to scrape
  • Use ephemeral debug containers (kubectl debug) for packet-level inspection with tcpdump
Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.