DNS is the dependency every other dependency goes through, and the reason it deserves a lesson rather than a paragraph is that almost nobody can describe the full path from a hostname in their code to an address in a socket. The path has at least six stages and each one caches, which means each one can be wrong independently.

What happens inside your process first

Before any packet is sent, the resolution is handled by a stub resolver in your process, and which stub you get depends on the language and the libraries, which is the first source of divergent behaviour.

A C program calling getaddrinfo goes through glibc, which consults /etc/nsswitch.conf to decide which sources to try in what order: files, meaning /etc/hosts, then DNS, possibly with other modules in between. A Go program by default uses its own pure-Go resolver, which reads /etc/resolv.conf directly and bypasses nsswitch entirely unless cgo is in use, which is why a Go binary and a shell command on the same host can resolve the same name differently. A JVM maintains its own cache in front of whatever the operating system does. Node uses the system resolver for lookup and a bundled resolver for dns.resolve, which behave differently.

The practical implication is that "the host resolves it fine" is not evidence that your application will, and the first step in any DNS investigation is to reproduce it from the same runtime rather than from dig.

Why the hosts file still matters

/etc/hosts predates DNS and remains the highest-priority source on most systems, which makes it both a useful tool and a persistent trap.

As a tool it is the fastest way to test a migration: point a name at the new address on one machine and exercise the whole application against it without touching any zone. It is also how most container images handle their own hostname, and how Kubernetes injects a pod's own name and any hostAliases.

As a trap, an entry added during an investigation and never removed produces a host that resolves one name differently from every other host in the fleet, and because no DNS tool queries the hosts file, the investigation that follows looks at caches and nameservers and finds nothing wrong. getent hosts is the command that reflects the real resolution order including files, which is why it beats dig as a first check when one machine disagrees with the rest.

The rule worth adopting is that any hosts-file entry added by a human gets a comment with a date and a reason, because the alternative is an unexplained line that nobody will dare delete two years later.

Where the stub sends its query

/etc/resolv.conf holds the configuration that matters: up to three nameserver entries tried in order, a search list of domains appended to unqualified names, and options including ndots, timeout and attempts.

The defaults are worth memorising because they produce real incidents. timeout defaults to 5 seconds and attempts to 2, so a single unresponsive nameserver listed first costs 10 seconds before the second is tried, which exceeds most request timeouts and appears as a total outage rather than a DNS problem. ndots defaults to 1, meaning a name containing at least one dot is tried as-is first, and anything in the search list is appended only afterwards.

cat /etc/resolv.conf
resolvectl status            # systemd-resolved's view, which may differ
getent hosts api.example.com # what nsswitch actually returns

On a desktop or a systemd host there is frequently a local caching resolver at 127.0.0.53, which means resolv.conf points at a local service whose own upstream configuration is elsewhere, and reading only resolv.conf tells you nothing about where queries actually go.

Recursion, and who does the work

The resolver your stub queries is almost always a recursive resolver, which accepts a question and takes responsibility for answering it completely. Your stub asks once and waits; the recursive resolver may send a dozen queries to get there.

Its first query, if it knows nothing, goes to a root server, of which there are 13 addresses served by hundreds of anycast instances. The root does not know api.example.com, but it knows which servers are authoritative for com, and it returns a referral naming them. The resolver then asks a com server, which returns a referral to the servers authoritative for example.com. The resolver asks one of those, which is authoritative and returns the answer.

That is iterative resolution, and the distinction from recursion is exactly who holds the task: a recursive resolver performs iterative queries on your behalf. Authoritative servers do not recurse; they answer for their own zones and refer you elsewhere otherwise.

Glue, and the chicken-and-egg problem

A referral names servers by hostname, such as ns1.example.com, and a resolver given only a name would have to resolve that name, which may itself be inside the zone it is trying to reach. The way out is glue records: the parent zone includes the nameservers' addresses alongside their names in the referral.

This is why registrar configuration asks for both nameserver names and, sometimes, their addresses, and why changing a nameserver's IP address requires updating glue at the registrar rather than only in the zone. Stale glue is a classic prolonged outage: the zone is correct, the authoritative server is running, and resolvers cannot find it because the parent is still handing out an address that no longer answers.

What the resolver has to track while it works

It is worth appreciating how much state a recursive resolver holds for a single question, because it explains both the speed of a warm resolver and the failure modes of a cold one.

For an uncached name the resolver must choose a root server from its hints, parse a referral, choose one of the TLD servers named in it, parse another referral, and choose one of the zone's servers. Each choice is an opportunity for latency, and good resolvers track the measured response time of every nameserver they know and prefer the fastest, which is why resolution latency for a popular name converges to the best available server over time while a rarely queried name pays the cost of discovery.

A resolver also enforces limits to protect itself: a maximum number of queries per client question, a cap on CNAME chain length, and a bound on how many times it will follow a referral. These matter because a maliciously or accidentally constructed zone can make resolution expensive, and the limits mean a deeply chained name may simply fail rather than resolve slowly. Chains of several CNAMEs, which accumulate easily when a name points at a CDN that points at another service, are the usual way to meet those limits unexpectedly.

Caching at every stage, with different lifetimes

Every stage on the path caches, and each honours the record's TTL in its own way, which is why a change appears to take effect at different times for different observers.

The recursive resolver caches the final answer for its TTL, and also caches the referrals, so the root and TLD lookups happen rarely. A local caching resolver on the host caches again. The language runtime may cache: older JVM configurations cached positive answers indefinitely, which is the origin of a long tail of applications that never notice a DNS change. And the application itself effectively caches by holding open connections, which is the most commonly overlooked layer because no DNS tool can see it.

Negative answers cache too, governed by the SOA record's minimum field rather than by any TTL on the missing record, which is why a name created after someone queried it too early can stay missing for longer than expected.

The transport, and when it stops being UDP

A query and its response travel over UDP port 53 by default, which as the UDP lesson covered is the right choice for a single small exchange. The size limit bites: a response too large for the negotiated buffer sets the truncated flag, and the client retries the same query over TCP port 53.

EDNS0 lets the client advertise a larger buffer, commonly 1,232 bytes, which is chosen to stay below the fragmentation threshold on most paths. Larger advertised buffers invite fragmentation and its associated loss, which is why the recommended figure came down over time. A DNS server reachable only on UDP, or a firewall permitting UDP 53 and blocking TCP 53, works until a response grows past the limit, which DNSSEC signatures and large record sets make routine.