When Failover Is What Actually Causes the Outage

The cleanest multi-region failure I have watched did not look like a failure for the first ninety seconds. One region got slow. Health checks marked it degraded. The global load balancer pulled traffic away from it. The dashboards showed traffic shifting cleanly into the other three regions. Everyone exhaled.

Then the survivors caught fire.

This is the pattern people miss. Failover is not recovery. Failover is a traffic move. Whether it produces recovery depends entirely on whether the destinations can hold the new load. If your regions were each running at 60 percent of peak, losing one of four means the survivors now have to handle roughly 80 percent each. That can work if every layer of the stack agrees with the math. It usually does not.

A few reasons the math breaks.

Cold caches. The failed region's working set lived in its local cache tier. When that traffic lands in another region, every read is a cache miss for the first minutes. Database read load can spike 5 to 10 times steady state. Replica lag grows. Some reads return stale or time out. The survivors start degrading too.

Autoscale lag. The autoscaler reacts to load. It needs minutes to read metrics, decide, request capacity, boot instances, pass health checks, and warm up. Two minutes is forever when the traffic shift takes ten seconds. You spend that gap running hot and feeding retries back into the system.

Capacity that does not actually exist. You provisioned three regions for the loss of one. The cloud provider had a bad day and that capacity is now committed to other customers doing the same thing. Reservations matter precisely because failover is when everyone wants the same hardware.

Client stickiness. DNS TTLs are 60 seconds at best, longer in misbehaving resolvers. Mobile clients cache resolved IPs across app launches. Long-lived gRPC and WebSocket connections do not redial until they break. A meaningful fraction of clients keeps talking to the bad region for many minutes, while clients that did move retry aggressively because their first attempts failed mid-transition.

The failure I want you to picture: a payments platform running four regions hot. One region's primary hit replication lag. The balancer correctly shifted writes to the next region. That region's connection pools were sized for steady state, not surge. They saturated in 40 seconds. Retries piled on. Inside three minutes, all four regions were over budget. The original incident lasted under a minute. The failover-induced outage lasted forty.

What actually makes multi-region resilient: real capacity reservation, warm cache prefill, load shedding before retry storms form, circuit breakers between regions, cell isolation so blast radius stops at a boundary, and graceful degradation when survivors are stretched.

Failover is the easy part. Carrying the shifted load is the work.