0%
Networking for Backend Engineers
Transport
Names and Trust
Application Protocols
Getting Traffic to a Server
Performance
Switching, ARP, and the Local Segment
A switch has no addresses, no routing table and no opinion about IP. It has one data structure and one rule, and almost every surprising behaviour on a local segment, including several that look like application bugs, comes from the way that data structure is maintained.
The MAC address table, learned by watching
A switch maintains a table mapping MAC addresses to ports, and it populates it entirely by observation: for every frame that arrives, it records the source MAC against the port it arrived on. Nothing announces itself and no protocol is involved. A host that has never transmitted is unknown to the switch, which is why a silent host is unreachable until it speaks, and why ARP for it works, since the ARP request is a broadcast and the reply teaches the switch where the host is.
Forwarding is then one lookup. If the destination MAC is in the table, the frame goes out that one port, which is the entire value of a switch over a hub. If it is not, the switch floods the frame out of every port in the broadcast domain except the one it arrived on, because the alternative is dropping a frame for a host that exists but has not spoken. A broadcast or multicast destination is flooded by definition.
Entries age out, typically after 300 seconds of no traffic from that address. That timer is the cause of a specific and confusing behaviour: a connection idle for longer than the ageing time has its entry removed, so the next frame to it is flooded rather than switched, which costs a little capacity and, on a network with asymmetric paths or a firewall counting flows, can produce a measurable hiccup on the first packet after an idle period.
Unknown unicast flooding is a real cost
Flooding is correct behaviour and it is also a leak. Every frame for an unlearned destination reaches every host in the broadcast domain, where each NIC discards it in hardware, so the bandwidth cost lands on every link rather than on one.
This becomes a problem in three situations worth recognising. Asymmetric traffic, where a host receives much more than it sends, such as a monitoring collector or a storage target, can have its entry age out between its own infrequent transmissions, so the heavy inbound stream is flooded continuously. MAC table exhaustion, where the table is full and cannot learn, degrades the switch towards hub behaviour, which is both a performance collapse and a security problem since every host then sees traffic for every other, and it is the mechanism of the classic MAC flooding attack. And virtual addresses that move, where traffic follows the old port until something teaches the switch otherwise, which is why failover software broadcasts gratuitous ARP.
Three destination kinds, and multicast is the awkward one
A destination MAC is one of three things, and a switch treats each differently. A unicast address identifies one interface and is forwarded or flooded as described. A broadcast address, all ones, is flooded to the whole domain by definition. A multicast address, identified by the low bit of the first byte being set, is addressed to a group, and the default behaviour is to flood it exactly like a broadcast because the switch has no idea who is interested.
That default is the problem. A single multicast stream of 100 Mbit/s, which is unremarkable for market data, video or a clustering protocol's heartbeat, is delivered to every port in the domain whether or not anything there wants it, so it consumes 100 Mbit/s of every link and costs CPU on every host whose NIC accepts the group.
IGMP snooping is the fix: the switch watches for IGMP membership reports from hosts joining a group and forwards that group's traffic only to the ports that asked. It needs an IGMP querier somewhere on the segment, usually the router, to prompt periodic reports, and this is the detail that breaks it. A VLAN with snooping enabled and no querier sees membership state time out, after which behaviour depends on the vendor: some fall back to flooding, which is wasteful but works, and some stop forwarding the group entirely, which silently breaks every multicast application on the VLAN. If a clustering product, a discovery protocol or a video stream works on one VLAN and not another, the querier configuration is the first thing to check.
Store-and-forward, cut-through, and where latency comes from
A store-and-forward switch reads the whole frame, validates its frame check sequence and then forwards it, so its latency includes the serialisation time of the frame: at 10 Gbit/s a 1,500-byte frame takes about 1.2 microseconds to receive, and that is added per hop. A cut-through switch begins forwarding as soon as it has read the destination MAC, around 500 nanoseconds in, so its latency is nearly independent of frame size.
The trade is error propagation. A cut-through switch has already sent most of a frame before it knows the frame was corrupt, so bad frames travel onward and are counted as errors at the first store-and-forward device, which can be several hops from the link that corrupted them. This is why error counters are read along a path rather than at its ends, and why low-latency trading networks, which are the main users of cut-through switching, accept error propagation in exchange for microseconds.
Neither mode changes throughput. Both forward at line rate on modern hardware, and the distinction only matters where per-hop latency is part of the product.
The buffers are small, and microbursts are what fill them
A switch port has a shallow output buffer, often a few hundred kilobytes shared across ports in a block, because fast memory is expensive. When arriving traffic for one port briefly exceeds its egress rate, the excess is buffered and then dropped.
The crucial word is briefly. Average link utilisation of 30 percent says nothing about whether the buffer overflowed, because congestion at a switch is a microsecond-scale phenomenon and monitoring systems report five-minute averages. A microburst, in which many senders transmit simultaneously towards one receiver, fills a port buffer in microseconds and drops frames while every graph on the dashboard looks idle. This is the normal mechanism behind the incast problem: a distributed query fans out to 40 servers, all 40 reply at once, and the switch port facing the requester cannot absorb 40 simultaneous responses.
The symptoms are TCP retransmissions and tail latency with no visible utilisation, and the evidence is on the switch rather than the hosts: per-port discard counters, which are the only place a microburst leaves a trace. Mitigations are architectural rather than clever, consisting of limiting fan-out, staggering responses, raising buffer allocations where the hardware allows it, and using congestion signalling such as ECN so that senders slow down before the buffer overflows rather than after.
Average utilisation is not evidence about congestion. A 10 Gbit/s port carrying 3 Gbit/s on a five-minute average can be discarding frames every second, because the buffer is sized in microseconds of traffic and the graph is sized in minutes. The only trustworthy evidence of switch-level congestion is the per-port discard or output-drop counter, and the only trustworthy evidence of its absence is that counter being flat while the problem occurs.
Half duplex still exists, and it is always a fault
Full duplex, in which both directions transmit simultaneously, has been the norm since switches replaced hubs, and collisions are a property of the shared medium that switches removed. So a port reporting half duplex today is never a design choice; it is a failed autonegotiation, usually because one end was hard-coded to a speed and duplex while the other was left automatic, which makes negotiation fail and the automatic end fall back to half duplex.
The symptom is distinctive and badly misleading: throughput collapses to a fraction of the link rate, late collisions and CRC errors accumulate, and the problem is worse under load, which makes it look like congestion or an application fault. ethtool eth0 printing Duplex: Half on a gigabit link is the whole diagnosis, and the fix is to set both ends to autonegotiate rather than to match the hard-coded setting, because hard-coding one end is what caused it.
What a switch gives you, and what it does not
It is worth being explicit about the boundary. A switch gives you a learned, per-port forwarding path, hardware-rate forwarding, and a broadcast domain within which every host can reach every other with no configuration at all.
It does not give you loop prevention without a protocol, which is the subject of the third section and the reason a redundant cable can take down a whole segment. It does not give you reachability between broadcast domains, which requires a router. It does not give you any flow control that TCP can rely on, since the pause frames of the standard are rarely enabled and deadlock-prone when they are. And it gives you no visibility at all by default: the only ways to see traffic on a switched segment are a port mirror, a tap, or flow export, because the whole point of switching is that traffic you do not own does not reach you.