Everything so far assumed a packet reaches the right network. This section is about how that happens, because the mechanism explains a class of production behaviour that looks inexplicable from inside a single network.

The internet is networks, not hosts

The unit of internet routing is not a host or even a subnet, it is an autonomous system: a network under one administrative control with a consistent routing policy.

Each one has a number, and there are roughly a hundred thousand in use. Cloud providers, large content companies, internet service providers, universities and many enterprises each operate one or several.

Routing between them is a separate problem from routing inside them. Inside, a network uses an interior protocol and optimises for shortest path, because it controls everything and wants efficiency. Between networks, policy dominates: who pays whom, which traffic is allowed to transit, and which paths are preferred commercially rather than technically.

Prefixes, and what gets announced

Networks do not announce hosts, they announce prefixes: address ranges in a compact notation where the number after the slash says how many leading bits are fixed.

A prefix with a smaller number is larger, covering more addresses. An organisation that holds a large range may announce it as one prefix or split it into several, and that choice has consequences explored shortly.

Forwarding uses longest prefix match: when several announced prefixes contain a destination address, the most specific one wins. That single rule is the basis of both legitimate traffic engineering and the most damaging form of routing attack, because a more specific announcement overrides a less specific one everywhere it is accepted.

BGP, briefly and accurately

The Border Gateway Protocol is how networks tell each other which prefixes they can reach and by what path.

It is a path vector protocol: an announcement carries the prefix plus the list of autonomous systems the route traverses. A network receiving an announcement prepends itself and passes it on to its neighbours, subject to policy. The path list serves two purposes, giving a crude distance measure and preventing loops, since a network that sees itself in a path discards the announcement.

It is explicitly not a shortest path protocol and not a performance-aware one. It knows nothing about latency, bandwidth, loss or congestion. A three-hop path through a saturated link is preferred over a four-hop path through an empty one, because hop count and policy are the only inputs.

Relationships decide where traffic goes

The policy that dominates path selection comes from three commercial relationships, and knowing them makes otherwise strange routing legible.

A transit relationship is a customer paying a provider to reach everywhere. A peering relationship is two networks exchanging traffic between their own customers at no charge, because it benefits both. A customer relationship is the reverse of transit.

The resulting preference is near-universal: send traffic to customers first, since they pay for it; then to peers, since it is free; then to transit providers, since it costs money. This single ordering explains most paths that look geographically absurd, and it means a packet's route is chosen by billing arrangements at least as much as by topology.

Why a path can be geographically absurd

Concretely, this produces routes that surprise anyone expecting physical sense.

Traffic between two networks in the same city may traverse another continent, because the two have no local peering and their transit providers interconnect elsewhere. Traffic may leave a region, cross an ocean and come back. A path may be asymmetric, with the forward direction taking one route and the reverse another entirely, because each direction is chosen independently by a different set of networks.

Asymmetry matters for debugging. A traceroute shows the forward path only, so a problem on the return path is invisible to it, and round trip latency is the sum of two routes that may differ substantially. Measurements from both ends are the only way to see both.

Hot potato and cold potato

Within these relationships, a network still chooses where to hand traffic over, and the two conventions have names.

Hot potato routing hands traffic to the next network at the nearest exit point, minimising the distance carried on your own expensive backbone. It is the default because it is cheaper, and it pushes the carrying cost onto the receiving network.

Cold potato routing carries traffic on your own backbone to an exit near the destination, which costs more and gives better control over the path's quality. Large content networks do this deliberately, because their own backbone is better provisioned than the public path and the latency improvement is worth the cost.

This is why traffic entering a well-engineered network early tends to perform better, and it is a real part of why content delivery networks improve latency beyond simple proximity.

Convergence takes time

BGP is not fast. When a path changes, the news propagates network by network, each applying its own policy and timers.

Convergence after a withdrawal typically takes tens of seconds and sometimes minutes, during which different parts of the internet have different views and packets may loop or be dropped. Timers deliberately damp oscillation, trading responsiveness for stability, and route flap damping can suppress a flapping prefix for far longer.

The practical consequence is that BGP is a mechanism for structural changes, not for failover measured in seconds. Anything requiring fast failover needs a mechanism above it, which is part of why health-checked load balancing and DNS-level steering exist despite routing already being dynamic.

Why this matters from inside an application

It is reasonable to ask why an application engineer should care about any of this, and there are four concrete reasons.

It explains latency that proximity does not. Two users equally distant from a region can see very different latency because their networks reach it by different paths, and no amount of application work changes it.

It explains incidents with no internal cause: a route leak elsewhere can make a service unreachable for a subset of users while every internal metric is healthy. It explains why anycast and content delivery networks work, which is the subject of the rest of this lesson. And it bounds what is fixable, because a path chosen by other networks' policies is not something to debug in code, and recognising that quickly is itself the skill.

Inside against between, and why two protocols exist

It is worth being explicit about why networks run two different routing protocols rather than one.

Inside a network, the operator controls every router, trusts every announcement, and wants the shortest path. Interior protocols therefore flood full topology information and compute shortest paths over it, converging in seconds and optimising for efficiency. They scale to thousands of routers, not to the whole internet.

Between networks, none of those assumptions hold. There is no shared trust, no shared metric, and no willingness to reveal internal topology. What is needed instead is a way to express policy, to withhold information deliberately, and to scale to a hundred thousand participants. BGP exchanges reachability rather than topology, which is both why it scales and why it cannot optimise anything.

The size of the routing table

One shared cost deserves a mention because it constrains what individual networks may do.

Every router carrying full internet routes holds an entry per announced prefix, and the global table is around a million entries for the older protocol and growing. Each entry consumes expensive memory in forwarding hardware, and a table that outgrows a router's capacity causes that router to fail in unpleasant ways, which has happened at identifiable thresholds and caused widespread disruption.

This is why gratuitously splitting a prefix into many smaller announcements is considered antisocial: the cost is borne by every network in the world, not by the announcer. It is also why announcements below a conventional minimum size are widely filtered, which is the fact that makes the defensive advice about prefix size work.