istio-proxy closing long running TCP connection after 1 hour
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
If an Istio-managed TCP connection closes after almost exactly one hour, the first thing to suspect is idle timeout, not random packet loss. Istio's DestinationRule TCP connection pool settings include an idleTimeout, and the documented default for TCP connections is one hour when it is not set explicitly. In other words, the "one hour" detail is a strong diagnostic clue.
What the One-Hour Behavior Usually Means
A long-lived TCP connection is not the same as an active TCP connection. If no bytes are sent or received for a long period, Envoy may treat that socket as idle and close it.
That means a connection can stay open for hours if traffic keeps flowing, but still get closed around the one-hour mark if it is mostly silent.
So the first question is not "How old is the connection?" It is "Was the connection idle?"
Confirm the Idle Case First
Before changing mesh config, verify whether the disconnect happens only when the socket is silent. A quick test is to send heartbeats over the same TCP stream.
If a connection with periodic traffic survives beyond an hour while a fully quiet connection does not, you are almost certainly dealing with idle timeout behavior.
The Istio Setting to Check
For TCP traffic, the relevant knob is trafficPolicy.connectionPool.tcp.idleTimeout on a DestinationRule.
Setting idleTimeout: 0s disables the timeout. You can also set a longer explicit duration instead of disabling it entirely:
This is often safer than turning the timeout off globally.
When Keepalive Is Better Than Stretching Timeouts
Just because you can disable idle timeout does not mean you should. A silent socket still consumes resources, and extremely permissive timeout settings can make stale connections linger longer than necessary.
If the protocol supports it, application-level heartbeats are often the healthier solution. They prove the connection is still meaningful and help both sides detect broken paths earlier.
A lightweight heartbeat can be as simple as:
In practice, many production systems combine moderate timeout settings with explicit heartbeats rather than choosing one or the other exclusively.
Remember the Entire Path
Istio may not be the only component that can close the connection. The path may also include:
- The client application
- The upstream server
- An ingress or egress gateway
- A cloud load balancer
- Another proxy outside the mesh
If the connection still dies after exactly one hour even after you change the Istio DestinationRule, the real limit may live somewhere else in that path.
Useful Commands for Inspection
When debugging, inspect the sidecar or gateway actually handling the traffic:
These commands help you confirm whether the proxy received a reset, applied a timeout, or inherited behavior from another layer.
If the flow crosses a gateway, inspect that gateway too. It is easy to focus only on the sidecar attached to the workload and miss the actual closing point.
Roll Out Narrowly
Timeout changes should be scoped carefully. Apply them to the specific service pair or workload that needs them rather than changing mesh-wide defaults immediately.
Longer idle lifetimes increase resource retention, so the operational question is not only "Does this stop the disconnect?" but also "What new cost does this create?"
That is why targeted DestinationRule changes are usually preferable to broad global changes during initial debugging.
Common Pitfalls
One common mistake is calling a connection "long-running" when it is actually idle for most of its lifetime. That wording hides the real cause.
Another issue is increasing timeouts without checking whether periodic traffic would solve the problem more cleanly. Heartbeats are often the better engineering answer.
Developers also sometimes edit the wrong layer. A sidecar DestinationRule will not fix a one-hour idle timeout enforced by a load balancer or gateway outside that path.
Finally, do not forget that idleTimeout is about inactivity, not absolute age. An active connection and an idle connection are treated very differently.
Summary
- A TCP connection closing after about one hour in Istio usually points to idle timeout behavior.
- The relevant setting is
DestinationRule.spec.trafficPolicy.connectionPool.tcp.idleTimeout. - The documented default TCP idle timeout is one hour when it is not set explicitly.
- Heartbeats are often a better fix than disabling timeouts completely.
- Check every layer in the connection path before concluding that Istio alone is responsible.
Related reading
- Istio DestinationRule gives upstream connect error or disconnect/reset before headers
- Istio does not forward Authorization header
- ITMS-90809 Deprecated API Usage -- Apple will stop accepting submissions of apps that use UIWebView APIs
- ITMS-91053 Missing API declaration - Privacy
- Java - sending HTTP parameters via POST method easily
- Java 8 How do I work with exception throwing methods in streams?
- Java 8 Lambda-Streams, Filter by Method with Exception
- Java 8 Stream and operation on arrays

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.