0%
Networking for Backend Engineers
The Stack
Transport
Names and Trust
Application Protocols
Getting Traffic to a Server
Reading a Packet Capture
A capture is the only tool that shows what actually happened on the wire rather than what a library reported, and most captures are useless because of decisions made before the first packet was recorded.
Where you capture decides what you can conclude
The single most consequential choice is the vantage point, and it is made before any filter is written.
A capture on the client shows what the client sent and received. A capture on the server shows the same for the server. Neither alone can distinguish loss from delay, because a packet that is absent from the receiving capture might have been lost in the network or never sent at all.
Capturing simultaneously at both ends is what turns a capture from suggestive into conclusive: a packet present at the sender and absent at the receiver was lost in between, and the direction is known. For any persistent problem, two captures are worth more than a hundred attempts at one, and the extra effort is a few minutes.
Filters that keep the capture usable
An unfiltered capture on a busy host produces gigabytes in seconds and is unusable, so the filter is not an optimisation but a prerequisite.
The filter language compiled into the kernel is the one to use for capture, because it discards unwanted packets before they are copied, which both reduces the file and avoids dropping packets under load. Filtering afterwards in a graphical tool uses a different, richer language that operates on already-captured data.
The useful filters are narrow and few: a host and port pair for one conversation, a port alone for a service, a flag combination to see only handshakes and resets, and a size threshold to exclude bulk data while keeping control packets. Writing the filter from the question being asked, rather than capturing broadly and hoping, is the habit that distinguishes a productive capture session.
Snapshot length and why headers are often enough
By default a capture records the whole packet, and that is frequently the wrong choice.
For a transport-level investigation, loss, retransmission, window behaviour, resets, handshakes, the headers are all that matter, and limiting the snapshot to the first hundred or so bytes reduces the file by an order of magnitude or more. That makes a long capture feasible, which matters because intermittent problems need long captures.
Record full packets when the payload is the question: a malformed response, an unexpected header, a protocol-level error. For encrypted traffic the payload is unreadable anyway, so truncation costs nothing. Deciding this deliberately is what makes the difference between a capture you can leave running for an hour and one that fills the disk.
Capturing for a long time without filling the disk
Intermittent problems require waiting, and waiting requires rotation.
A ring buffer of a fixed number of files of a fixed size keeps the most recent window and discards the rest, so a capture can run for days within a bounded footprint. Combined with header-only truncation and a narrow filter, a week of a specific conversation fits comfortably.
The refinement that makes this genuinely useful is a trigger: run the capture continuously and have whatever detects the problem stop it, so the buffer holds the window around the event rather than whatever happened to be last. Without a trigger, the usual outcome is a capture that rotated past the incident by the time anyone looked.
Drops in the capture itself
A capture can lose packets, and a capture that silently lost packets produces false conclusions.
When the capture buffer fills faster than the tool drains it, packets are dropped before being written, and the tool reports the count when it exits. That number must be checked, because dropped packets look exactly like network loss in the resulting file, and the resulting investigation chases a problem that does not exist.
The remedies are a narrower filter, a shorter snapshot, a larger buffer, and writing to a fast local disk rather than a network filesystem. A capture reporting any drops should be redone rather than analysed, which is why checking that number is the first thing to do with a finished capture.
Capturing in containers and behind proxies
Modern deployments add layers, and each one changes where a capture must be taken.
A container shares its network namespace with its pod or is isolated in its own, so capturing on the host sees traffic on the host's interfaces and may or may not see the container's. Entering the namespace is the reliable approach, and it requires the capability to do so, which hardened images deliberately lack.
Where a sidecar proxy intercepts traffic there are effectively two conversations, the application to the proxy over the loopback interface and the proxy to its peer over the network, and they must be captured separately. Guessing which side a problem is on wastes the most time here, so capturing both sides of the proxy at once is the efficient move.
The tools and when each one fits
Three kinds of tool cover every situation and choosing wrongly costs time.
A command-line capture tool is what runs on a server: it is available everywhere, it costs almost nothing, and it writes a file for later analysis. This is the capture step and it is where the filter, the snapshot length and the rotation are decided.
A graphical analyser is where the reading happens. It reassembles conversations, decodes hundreds of protocols, flags retransmissions and out-of-order packets, plots throughput and round trip over time, and follows a stream as readable text. Pulling the file to a workstation and opening it there is almost always better than squinting at terminal output.
A scriptable analyser sits between them and is the right tool for a question asked across many captures or a very large one: counting retransmissions per connection, extracting every handshake's options, or finding connections that ended in a reset. Knowing all three exist, and that the capture and the analysis are separate steps, is most of the skill.
Permissions, and why capture is privileged
Capturing requires privileges that production systems deliberately restrict, and knowing the shape of that restriction saves an argument.
Reading raw packets needs a specific capability, because it exposes every other process's traffic on the host. Granting it to the capture binary rather than running the whole session as a privileged user is the usual compromise, and many hardened container images remove the capability entirely by design.
That restriction is reasonable, and the ways around it are to capture on the host rather than in the container, to run a dedicated capture container with elevated privileges in the same network namespace, or to capture at a mirror point on the network. Agreeing which of those is acceptable before an incident is considerably better than negotiating it during one.
What a capture cannot tell you
Setting expectations correctly prevents the most frustrating kind of investigation.
A capture cannot explain why an application was slow to send, only that it was. It cannot show which device in the middle dropped a packet, only that the packet was sent and not received. It cannot read encrypted payloads without keys. It cannot show traffic that never left the host, unless captured on the loopback interface. And it cannot distinguish a packet dropped by the capture mechanism from one dropped by the network, which is why the drop count matters.
Within those limits it is the most authoritative source available about what happened on the wire, and outside them it is actively misleading, so the boundary is worth knowing.
One capture is suggestive and two are conclusive. A packet missing from a single capture may have been lost in the network or never sent, and nothing in that file distinguishes the two; present at the sender and absent at the receiver proves loss and names the direction. Check the tool's reported drop count before analysing anything, because packets the capture itself dropped are indistinguishable from network loss and send the investigation after a problem that does not exist.