0%
Networking for Backend Engineers
The Stack
Transport
Names and Trust
Application Protocols
Getting Traffic to a Server
Packet Loss, Retransmission and Tail Latency
Packet loss is treated as a fault, and most of the time it is a normal and necessary part of how networks work. Understanding why it happens determines whether you should fix it, tolerate it or change how the application reacts to it.
Loss is the congestion signal
The internet has no mechanism for a router to tell a sender to slow down. What it has is a buffer that fills and then discards, and the discard is the message.
A sender that loses a packet concludes the path is congested and reduces its rate. That feedback loop is what keeps a shared network from collapsing, and it requires loss to function. A path with zero loss under load is a path where something else is limiting the rate, not a path that is better.
This is the single most important reframing in the topic. Loss in the one-in-ten-thousand to one-in-a-thousand range on a loaded path is the system working. Loss that is persistent, bursty, independent of load or concentrated in one direction is a fault, and distinguishing the two is most of the diagnostic work.
Buffer overflow
The dominant cause is the one just described, and its characteristics are worth naming so it can be recognised.
It is load-dependent, appearing when the link is busy and vanishing when it is not. It affects all traffic sharing the bottleneck, so other flows see it too. It is often bursty, because a full buffer discards several consecutive packets. And it is accompanied by elevated latency, since the buffer was full before it overflowed, which is the signature that distinguishes it from other causes.
Loss with elevated latency means congestion. Loss without elevated latency means something else, and that distinction should be the first thing checked.
Wireless and physical causes
On wireless links loss has an entirely different character, and treating it as congestion is the mistake that makes mobile performance bad.
Interference, distance, obstruction and competing transmitters cause frames to be corrupted or lost regardless of how much traffic is flowing. Wireless technologies retransmit at the link layer to hide this, which largely works, but it converts loss into delay and jitter, and when it fails the loss reaches the transport.
The crucial property is that this loss is independent of the sender's rate, so reducing the rate does not reduce it. A congestion control algorithm that interprets it as congestion will back off repeatedly for no benefit, which is why loss-based algorithms perform poorly on mobile networks and why algorithms that model the path do better there.
Physical faults, a damaged fibre, a failing optic, a bad cable, produce constant low-level loss unrelated to load, often with errors visible in interface counters, which is the fastest way to confirm them.
Policing and shaping
A rate limit implemented by discarding excess rather than queueing it produces loss with a distinctive shape.
A policer drops anything above a configured rate immediately, so loss appears abruptly when traffic crosses a threshold and the throughput curve clips flat. A shaper instead queues and delays, so the same limit appears as latency rather than loss. Both are configured limits rather than capacity limits, which means no amount of investigating the path will explain them.
The signature of policing is throughput pinned at a suspiciously round number with loss proportional to the excess. Recognising it saves hours, because the answer is a contract or a configuration rather than anything in the network.
Deliberate dropping by queue management
Modern equipment often drops before the buffer is full, on purpose, and this is an improvement rather than a fault.
Active queue management discards or marks a small fraction of packets as the queue grows, signalling congestion early so senders slow down before the buffer is deep. The result is a shorter queue, lower latency and loss that appears at moderate load, which can look alarming while being exactly the intended behaviour.
Explicit congestion notification does the same without discarding, marking packets instead so the receiver can tell the sender to slow down. Where it is supported end to end it gives the congestion signal without the retransmission cost, which is strictly better, and it is increasingly enabled by default.
Microbursts
A cause of loss that confuses people because the averages do not show it.
Traffic is not smooth. A ten gigabit interface averaging two gigabits can receive bursts at full line rate for microseconds, and if several sources burst simultaneously into one egress port, the buffer overflows despite a comfortable average. Monitoring that samples every thirty seconds sees twenty percent utilisation and reports no problem.
The signature is loss with low average utilisation, and it is confirmed with high-resolution interface counters or buffer occupancy metrics where the equipment exposes them. The remedies are more buffer at the right point, traffic smoothing at the senders, or spreading the traffic across more paths, and the first step is simply knowing this is possible.
Loss in one direction only
Because round trip measurements mix both directions, directional loss is easily misattributed.
A path can be clean outbound and lossy inbound, or vice versa, and the consequences differ. Loss of data packets triggers retransmission of data. Loss of acknowledgements can be tolerated, since a later acknowledgement covers earlier ones, so inbound acknowledgement loss is less damaging than it sounds, but heavy loss of them stalls the sender's window.
Determining the direction requires looking at the counters on each side or capturing at both ends, and it matters because the remedy, and often the responsible party, is different for each direction.
Loss inside a data centre
Internal networks are often assumed to be lossless, and they are not, which matters because the consequences there are proportionally worse.
The causes differ from the internet's. Incast is the characteristic one: many servers responding simultaneously to one requester converge on a single switch port whose buffer cannot hold the combined burst, so packets are discarded even though every link is far from saturated on average. This is the normal shape of a fan-out query, so it is designed into many systems unintentionally.
Shallow buffers on high-speed switching hardware make this worse, since fast ports with small buffers tolerate very little burstiness. And because internal round trips are measured in microseconds while the retransmission floor is measured in hundreds of milliseconds, a single internal loss costs proportionally thousands of times more than the same loss on an internet path.
Reordering is not loss, and is often mistaken for it
Packets taking different paths can arrive out of order, and a sender sees the same duplicate acknowledgements that loss produces.
Causes include load balancing across several equal-cost paths, which is standard in data centres and in provider backbones, and link aggregation that hashes on fields that vary within a flow. Well-behaved load balancing keeps a flow on one path precisely to avoid this, which is why the hash includes the port numbers.
The distinction matters because the remedies are different: reordering is addressed by consistent path selection, while loss is addressed by capacity or queue management. Reordering also wastes throughput by triggering unnecessary rate reductions, so a path with heavy reordering performs worse than its loss rate predicts, and a capture at both ends is what distinguishes the two conclusively.
What a sender cannot tell you
A limitation worth internalising: the sender knows a packet was not acknowledged, and nothing else.
It cannot tell whether the data packet was lost or the acknowledgement was lost, whether the loss was congestion or corruption, where along the path it happened, or whether the packet was merely delayed. All of those produce the same observation, and the response, retransmit and slow down, is the same.
This is why loss diagnosis requires information from outside the transport: interface counters, measurements from both ends, per-hop probing and comparison across host pairs. Expecting the endpoints to explain the loss is expecting information they do not have.
The strong answer treats loss as the congestion signal rather than as a defect: the network has no way to tell a sender to slow down except by discarding, so a loaded path with a small amount of loss is working as designed. The diagnostic question is which kind of loss it is, and the first discriminator is latency. Loss accompanied by elevated round trip time is congestion; loss with normal latency is wireless, physical, policing or a microburst, and reducing the sending rate will not help those at all.