Diagnosing Connection Failures

Topics Covered

The Elimination That Comes First

The five things that must all work

The three error messages and what each proves

Testing resolution separately from connecting

The connection test that answers two questions

Confirming what is listening

Where to run each test

Protocol-level failures that look like connection failures

Timeouts that are yours rather than the network's

What a working client proves

The question worth asking before any of it

A connection failure has a small number of possible causes and they are arranged in a sequence. Testing them in order resolves most failures in minutes; testing them in the order they occur to you takes a day.

The five things that must all work

A connection requires name resolution, a route, permission at every filter, something listening, and an application willing to respond. A failure is one of those five.

They are strictly ordered, which is the useful part: there is no point examining firewall rules before confirming the name resolved to the address you expect, and no point reading application logs before confirming anything is listening. Each step either passes or names the problem.

The sequence also maps onto the tools. Resolution is checked with a resolver query, the route with a path probe, the filter and the listener together with a connection attempt, and the application with a protocol-level request. Four commands cover the whole space.

The three error messages and what each proves

Before any tool, the error itself eliminates most of the possibilities, and this is the highest-value thirty seconds available.

Connection refused means a reset arrived. Something at that address responded actively, so resolution worked, the route works in both directions, and no filter dropped the packet. The remaining causes are nothing listening on that port, a full accept queue configured to reject, or a filter configured to reject rather than drop. This is the most informative failure because it eliminates the most.

Connection timed out means silence. The packet went somewhere and nothing came back, which means a filter dropping rather than rejecting, a route to nowhere, an address nothing answers at, or a return path that fails. Resolution worked, but nothing beyond that is established. A name resolution error means the failure happened before any packet was sent at all.

The five requirements are strictly ordered, and the error message alone establishes how far down the sequence the connection reached.

Testing resolution separately from connecting

Combining the two is what makes failures confusing, so separate them immediately.

Query the resolver directly for the name and read the answer: does it resolve at all, what address or addresses does it return, and are they the addresses you expect. A name resolving to an old address is one of the most common causes of a failure that makes no sense, and it is invisible if you only ever try to connect.

Then connect to the address rather than the name, which removes resolution from the picture entirely. If the address works and the name does not, the problem is resolution, and that is a complete diagnosis. If both fail identically, resolution is fine and the problem is further along.

The connection test that answers two questions

A single connection attempt to an address and port distinguishes a filter, a route and a missing listener.

A tool that opens a connection and reports the outcome, with a short timeout, gives one of the three outcomes above, and the timing matters too: an immediate refusal means a reset came back promptly, while a refusal after a second or two means something slower was involved.

Run it from the host that is failing, because running it from your workstation tests a different path through different filters and proves nothing about the failing case. This is the single most common methodological error in connection debugging: testing from the wrong place and concluding the service is fine.

Confirming what is listening

On the server side, one question settles whether the failure is network or application: is anything bound to that address and port.

A socket listing shows the listening sockets with their bound addresses, and the detail that matters is the address rather than the port. A process bound only to the loopback address is unreachable from anywhere else no matter how correct every firewall rule is, and this is a frequent cause of a service that works locally and nowhere else.

A process bound to a specific address rather than all addresses is reachable only on that address, which breaks when the host has several or when traffic arrives on an unexpected one. Reading the bound address first saves examining the network for a problem that is a one-word configuration change.

Where to run each test

The vantage point determines what a test means, and this deserves to be explicit.

Test from the failing client, because that is the path in question. Test from another client that works, because the difference between them localises the problem. Test from the server to itself over the loopback interface, which proves the application is alive independently of the network. Test from the server to its own external address, which exercises the interface and the local filter.

Those four vantage points, with the same test, partition the problem precisely: working locally but not from the server's own address means a local filter or binding; working from the server but not from another host means the network between them; working from one client and not another means something specific to that client's path.

Protocol-level failures that look like connection failures

A connection that establishes and then fails is a different problem, and distinguishing it early prevents examining the network for no reason.

A certificate error, an authentication rejection, a protocol version mismatch and an application error all require a working connection first, so they prove the entire network path is fine. The error text usually says so, but it is easy to read any failure as a connectivity problem when the service is simply unreachable in the user's words.

The clean test is whether a plain connection to the address and port succeeds. If it does, the network is eliminated completely and the remaining work is in the protocol or the application, which is a different team's problem and a different set of tools.

Timeouts that are yours rather than the network's

A client's own timeout configuration shapes what a failure looks like, and that is worth knowing before interpreting one.

A connect timeout shorter than the handshake needs on a long path produces failures that look like unreachability but are impatience. A timeout below the retransmission timer's floor means a single lost handshake packet always fails rather than recovering, which turns a tiny loss rate into a visible error rate.

So when a failure is a timeout, the first question is how long the client waited, and whether that is long enough for the path in question. Elsewhere the opposite error dominates: a thirty second connect timeout means a dead dependency ties up a request thread for thirty seconds, which turns one broken dependency into an exhausted thread pool.

What a working client proves

Comparing against something that works is the fastest available diagnostic and it is frequently skipped.

If another host connects successfully, the service is up, the listener is bound, and a path exists, so the failure is specific to the failing client's path or configuration. The investigation then becomes a comparison: what differs between these two hosts in subnet, security group, resolver configuration, image version or configuration.

If no host can connect, the problem is at the destination or is general, and the comparison is across time instead: what changed. Establishing which of those two situations you are in should happen in the first two minutes, because the subsequent work is entirely different.

The question worth asking before any of it

A surprising share of connection investigations begin without a specific failing case, which makes them unanswerable.

"The service is flaky" cannot be diagnosed. "This client instance, connecting to this address and port, at this time, received this error" can be, and getting from the first statement to the second is often the whole of the hard part. The fields needed are a client, a target, a timestamp and the exact error text.

When those are not available, obtaining them is the first task rather than an inconvenience, and it is usually a logging change rather than an investigation. Starting without them produces a day of speculation and no conclusion, which is the most common shape of a wasted incident.

Level Expectations

The error message is worth thirty seconds before any tool. Refused proves a reset arrived, so resolution, routing in both directions and every filter already worked, leaving only a missing listener or a deliberate rejection. Timed out proves silence, so something dropped the packet and nothing beyond resolution is established. Running the same test from the failing client, a working client, the server's loopback and the server's own external address partitions the remaining space precisely, and testing from your workstation instead of the failing host proves nothing at all.