Almost every packet in a production system has its addresses rewritten at least once before it arrives. Knowing where, and what the rewrite does to the information you need, is the difference between reading a log correctly and reading it wrongly.

The translation table

Network address translation is a device rewriting addresses and ports in packet headers and remembering what it rewrote so it can reverse the change on the way back.

The memory is the important part. A translating device holds a table keyed by the rewritten tuple, mapping it back to the original one. When a response arrives addressed to the rewritten tuple, the device looks it up, restores the original addresses, and forwards it. Without that entry the response cannot be delivered, which is why translation is inherently stateful and why its table size is a capacity limit.

That single fact explains most of the behaviour in this lesson: translation requires state, state has a size, state has an expiry, and when the state goes away the connection silently stops working.

Source translation and destination translation

The two directions have different purposes and different failure modes.

Source translation rewrites the source address of outbound packets, so many internal hosts appear to the outside as one address. This is what a home router does, what a cloud NAT gateway does for private subnets, and what a Kubernetes node does for pod traffic leaving the cluster. Its purpose is to let hosts without routable addresses reach the internet.

Destination translation rewrites the destination address of inbound packets, so a published address is mapped to an internal one. This is what a layer 4 load balancer in network address translation mode does, and what port forwarding on a router does. Its purpose is to let outside clients reach hosts they cannot address directly.

Many paths include both: a request arrives, is destination-translated to a backend, the backend calls out, and that call is source-translated on the way to its dependency.

What the port is doing

When many hosts share one address, the port is what distinguishes their connections, which is why the mechanism is more precisely called network address and port translation.

The device picks an unused source port on the shared address for each outbound connection and records the mapping. Two internal hosts both using source port 50000 get different external ports, so return traffic is distinguishable.

This is the origin of the capacity limit that bites hardest in practice. There are at most 65,535 ports, and in practice around 28,000 to 64,000 usable ones per external address, per destination tuple. A translating device fronting many hosts talking to one popular destination can run out, and the failure is a refusal to establish new connections while existing ones work fine.

Why the original address disappears

From the receiving end's point of view, the consequence is simple and frequently inconvenient: the source address on the packet is the translator's, not the client's.

So a backend behind a translating load balancer logs the balancer's address for every request. Rate limiting by source address limits the balancer rather than the client. Geolocation reports the balancer's location. Allowlists based on client address match nothing useful. Abuse controls that count per address see one address.

None of this is a bug, and all of it is a design problem to solve explicitly. The solutions are the subject of a later section: the proxy protocol at layer 4, and X-Forwarded-For or Forwarded at layer 7.

Hairpinning and the loop that fails

A specific and confusing case arises when an internal host reaches a service by its external address.

The packet leaves the host destined for the public address, arrives at the translator, is destination-translated to an internal backend, and the backend replies directly to the internal client because they are on the same network. The client receives a reply from an address it never sent to, and discards it, so the connection hangs.

Making this work is called hairpinning or NAT loopback, and it requires the translator to also source-translate so the reply comes back through it. Many devices do not do this by default. The practical lesson is that internal clients should reach internal services by internal names, and a service that works from outside but hangs from inside almost always has this shape.

Timeouts, and the silent death of idle flows

Every translation table entry has an idle expiry, and the expiry is the single most common cause of mysteriously hanging connections.

When an entry expires, the device forgets the mapping. Both endpoints still believe the connection exists, because nothing told them otherwise. The next packet in either direction arrives at the translator with no matching entry and is dropped, usually silently. The sender retransmits, gets nothing, and the application hangs until its own timeout fires, which may be minutes or may be never.

Typical idle timeouts are a few minutes for TCP on cloud NAT gateways and shorter for UDP. The defence is keepalive traffic at an interval below the shortest timeout in the path, which is why TCP keepalive intervals of a few minutes are standard and why database connection pools need their own idle validation.

Where translation happens in a real path

It is worth enumerating the translating hops in a typical cloud request, because people usually count fewer than exist.

The client's home or mobile network translates on the way out, often behind carrier-grade translation shared by thousands of subscribers. The cloud provider's load balancer may destination-translate to a backend. In Kubernetes, the service implementation destination-translates the service address to a pod address, and source-translates when the traffic crosses nodes or leaves the cluster. A sidecar proxy redirects traffic locally, which is another translation. Outbound calls traverse a NAT gateway that source-translates to a shared address.

That is four or five rewrites, each with its own table, its own limits and its own timeout. When an address in a log does not match an address you expect, the useful question is which of these hops you are looking at the output of.

Why translation exists at all

It helps to remember that none of this is a design anyone wanted, and that two separate pressures created it.

The first was address scarcity. There are about four billion addresses in the older protocol, which ran out, and translation let many hosts share one routable address, which deferred the problem for decades. The second was architectural: private address ranges are not routable from the internet, so a host in one cannot be reached without something publishing an address on its behalf.

The newer protocol has enough addresses that sharing is unnecessary, and translation for scarcity is not needed there. Even so it persists, partly because private ranges remain a convenient administrative boundary and partly because load balancing and service abstraction use the same rewriting mechanism for reasons unrelated to scarcity. So translation is not going away with address exhaustion, and the behaviours in this lesson remain relevant regardless of protocol version.

Connection direction and who can initiate

A consequence of source translation that shapes architecture is that it makes connections one-directional.

A host behind source translation can initiate outbound connections, because the act of sending creates the table entry that return traffic needs. Nothing outside can initiate inbound, because no entry exists and the translator does not know which internal host a new connection is meant for.

This is why services behind translation need something publishing them, and why anything wanting to push to a client uses a connection the client opened: long polling, server-sent events, WebSockets and mobile push all exist partly because of this constraint. It is also why peer-to-peer connectivity requires hole punching, where both peers send outbound to create entries and a coordinating server tells each the other's translated address.