Introduction

When HTTP requests look “delayed” in Kubernetes on AWS, the delay is usually not a component intentionally holding packets. More often it is queueing, connection reuse behavior, health-check transitions, overloaded pods, or node-level network pressure. kube-proxy and ELB are part of the traffic path, but you need to measure each hop before blaming either one.

Understand the Request Path First

A typical path is:

1. client connects to an AWS load balancer
2. the load balancer forwards to a node or directly to a target, depending on setup
3. Kubernetes service routing sends traffic to a pod
4. the application handles the request

A delay at any of those layers can feel identical from the client side.

That is why “ELB is delaying packets” and “kube-proxy is delaying packets” are usually hypotheses, not conclusions.

What kube-proxy Actually Does

kube-proxy programs service routing rules, commonly through iptables or IPVS. It is not an application proxy buffering HTTP bodies at Layer 7. Its job is to translate service traffic to backend pod endpoints.

That means typical Kubernetes-side causes are:

• a large or stale conntrack table
• overloaded nodes
• endpoint churn while pods are starting or failing
• uneven traffic distribution
• readiness probes marking pods unavailable too slowly or too late

In iptables mode, rules are kernel-level packet filters. In IPVS mode, service load balancing is often more efficient under larger endpoint counts, but either mode can still look slow if the node or pods are unhealthy.

What the AWS Load Balancer Might Be Doing

On AWS, the load balancer type matters:

• ALB works at Layer 7 and can show request-level timing metrics
• NLB works at Layer 4 and is closer to raw TCP forwarding
• Classic ELB is older and behaves differently from both

Perceived delay can come from:

• targets failing health checks and traffic being retried elsewhere
• idle timeout mismatches
• backend connections not being accepted quickly
• cross-zone traffic or target scarcity during scaling events

Again, this is usually system behavior under conditions, not a hidden “delay packets” feature.

Measure Before You Guess

Start with observations at each layer.

Check Kubernetes endpoints and pod readiness:

1kubectl get svc my-service
2kubectl get endpoints my-service -o wide
3kubectl get pods -o wide
4kubectl describe pod my-pod

Check whether the node is under connection-tracking pressure:

sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max

Check the application response time directly from inside the cluster:

kubectl run curlbox --rm -it --image=curlimages/curl -- \
  curl -s -o /dev/null -w 'connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}\n' \
  http://my-service.default.svc.cluster.local/health

If in-cluster timing is already high, the problem is likely not the external load balancer.

Common Real Causes

In practice, these issues show up more often than a kube-proxy bug:

• application threads are saturated, so requests wait before processing
• readiness probes allow traffic before the app is actually warm
• a few pods are hot while others are idle
• conntrack tables are near capacity on busy nodes
• keep-alive, idle timeout, or retry settings are mismatched between client, load balancer, and app

Notice that none of those require packet delay as an explicit feature. They create delay as an emergent effect.

When To Suspect the Networking Layer

Only after you rule out application latency should you focus on the networking layer itself. Stronger indicators include:

• retransmissions visible in packet captures
• sudden spikes tied to node changes or service rule updates
• clear improvement when switching service mode or node type
• cluster-wide symptoms across multiple unrelated services

At that point, compare node metrics, load balancer metrics, and app timings side by side.

Common Pitfalls

• Calling any latency symptom a kube-proxy or ELB delay without measuring the application first.
• Ignoring readiness and health-check behavior during deployments.
• Looking only at client-side latency and not at in-cluster timings.
• Forgetting that ALB, NLB, and Classic ELB have different behavior and metrics.
• Assuming kube-proxy is an HTTP-aware component when it usually is not.

Summary

• 'kube-proxy and ELB are part of the request path, but neither should be blamed without measurements.'
• Most “packet delay” reports are really queueing, retries, health-check churn, or backend slowness.
• Measure service endpoints, pod readiness, conntrack pressure, and in-cluster latency first.
• Load balancer type matters because ALB, NLB, and Classic ELB behave differently.
• Treat the problem as end-to-end latency analysis, not as a search for one guilty component.