Post

System Design

The Microsoft 365 Outage, Explained: 5 System Design Lessons

On November 12, 2026, Outlook and Exchange Online went down worldwide for almost 24 hours. A bug that had been running for three months, a routine failover and a capacity dip lined up. What happened, why recovery took so long, and the design lessons behind it.
By Codemia • November 20, 2026

9 min read

On Thursday, November 12, 2026, at 16:20 UTC, Exchange Online stopped working for a large part of the world. Outlook could not connect, webmail failed, and Teams, Copilot, SharePoint and Defender lost every feature that leans on Exchange. Global webmail did not come back to normal until 14:17 UTC the next day, and Microsoft did not close the incident until 16:10 UTC, almost 24 hours after it started.

There was no attack and no bad deploy that day. The change that broke it had been running in production for more than three months. What turned it into a global outage was a routine traffic shift, a capacity dip that the system was designed to absorb, and safety mechanisms that made things worse.

That makes it one of the best real-world system design case studies of the year. Every piece of it maps onto a question you could be asked in an interview: shared dependencies, headroom, load-triggered bugs, retries and failover, bulkheads, and recovery from overload.

Sources

This post is based on Microsoft's preliminary Post Incident Report (EX1550543 and MO1550821, dated November 15, 2026), as reported by IncidentHub, 365Fanatic, BleepingComputer and The Register. Microsoft's final report is due November 20, 2026, and may change details. Figures here are the preliminary ones.

What happened, in one paragraph

Microsoft 365 traffic enters through a fleet of networking gateway servers. In July 2026, Microsoft shipped a fix to those servers that contained a hidden defect: under a particular load pattern, an internal Windows process hit lock contention, burned through CPU and memory, and the server stopped responding. On November 11, an unrelated problem took nearly 14% of the gateway's capacity out of service. On November 12 at 01:07 UTC, Teams automatically failed over from its alternative gateway onto the primary gateway, the one Exchange Online also uses. Load climbed all day. When the US business day peaked, servers started hitting the defect, traffic management moved their users onto healthy servers, and those servers hit it too. In under two hours, more than 80% of the gateway fleet was unresponsive.

Figure 1. No single factor caused the outage. A dormant bug, a capacity dip the system could absorb, and a routine failover each looked harmless alone.

The timeline

All times are UTC, from the preliminary report and Microsoft's status updates.

TimeWhat happened
July 2026Gateway fix with the latent defect rolls out over several weeks through safe deployment
Nov 11Nearly 14% of gateway capacity goes out of service for an unrelated reason
Nov 12, 01:07Teams fails over onto the primary gateway shared with Exchange Online
Nov 12, 16:20Incident starts; Exchange Online alert EX1550543
Nov 12, 18:27Second notice MO1550821 for Teams, Copilot, Defender and others
Nov 12, 19:00All deployments and configuration changes paused
Nov 12, 19:20Engineers find the load comes from unavailable servers, not more traffic
Nov 12, 23:55Restored servers stay healthy briefly, then fail again under traffic
Nov 13, 00:30Connection negotiation identified as the failing path
Nov 13, 07:20Fix reverting the defective code path applied to part of the fleet
Nov 13, 11:35Asia Pacific and Europe recovered
Nov 13, 14:17Global webmail back to expected levels
Nov 13, 16:10Incident ends; email backlog drains afterwards

Notice the gap between 01:07 and 16:20. The trigger had already happened fifteen hours before anyone noticed. The system ran hot all day, and it only broke when daily peak traffic arrived on top.

Lesson 1: A shared dependency is a shared fate

Teams had its own alternative gateway. Exchange Online did not. When Teams failed over, it moved onto the gateway that Exchange depended on, so a protective action for one service quietly became extra load for another.

The report also says Teams, OneDrive, SharePoint and the Office apps could fail away from the broken gateway and recovered sooner. Exchange Online could not, "for scale reasons," so it stayed down the longest.

In an interview: when you draw a shared component, such as an API gateway, a load balancer tier, an auth service or a message bus, ask what happens to everyone else when one tenant fails over onto it. Good answers mention per-tenant quotas at the shared layer, separate pools for the most critical services, and checking that the failover target actually has room for the traffic it is about to receive.

Lesson 2: Headroom is not a number, it is a budget

Losing 14% of capacity was fine on its own. The gateway had headroom for that. Absorbing a Teams failover was also normal operations. What nobody checked was whether there was headroom for both at once, at US peak.

Capacity planning usually asks "can we lose one zone?" or "can we absorb one failover?" Real incidents spend the budget twice. Before you shift traffic onto a fleet, the question is how much headroom is left right now, after everything else that is already broken.

In an interview: state your headroom assumption explicitly (for example, run at 50 to 60% utilization at peak so you can lose a zone), then say what consumes it: lost capacity, failovers, retries and peak traffic. Gate automatic failover on current spare capacity, not on the target being "healthy."

Lesson 3: Load-triggered bugs hide from normal testing

The defect shipped in July and ran worldwide for more than three months without impact. Safe deployment, flighting and canaries all passed, because the bug only appears under a specific load and traffic mix. According to the report, load testing missed it because the labs used a traffic mix and hardware that did not match production, and the lock contention is worse on newer servers with more CPU cores.

This is why "it has been fine in production for months" is weak evidence. A canary at average load cannot find a bug that only shows up at peak load on your biggest machines.

In an interview: when you describe a rollout, mention load testing at or above production peak, on production-like hardware, and with a realistic traffic mix. Mention that a slow rollout reduces blast radius but does not prove safety under conditions the fleet has not yet seen.

Lesson 4: Failover can spread a failure instead of containing it

When a gateway server stopped responding, traffic management did what it was built to do: it moved users onto healthy servers. But the failure was caused by load, so moving load onto the remaining servers pushed them over the same edge. Each failure made the next one more likely. At 19:20 UTC on November 12, engineers confirmed that total traffic had not gone up at all. The load on each surviving server came from the servers that had already died.

Figure 2. A cascading failure. Total traffic stays flat, but every server that dies hands its users to the survivors, until more than 80% of the fleet is unresponsive.

The gateway was supposed to stop this. It is split into thousands of "capacity units" grouped into regions, which act as bulkheads: a failure in one unit or region should stay there. The report says a separate issue prevented that containment, so the failure spread globally. Microsoft has not yet explained that issue; the final report should.

Figure 3. Bulkheads only work if traffic is not allowed to cross them when things go wrong. Here the failure crossed every region.

In an interview: pair every failover mechanism with a limit. Load shedding (rejecting some requests early with a cheap error so the rest succeed), per-server admission control, circuit breakers and a cap on how much traffic a region can import from another are all ways to say "fail some users cleanly instead of failing everyone slowly."

Lesson 5: Recovering from overload is harder than the failure

Engineers started restarting servers on the evening of November 12. It did not work. A restarted server came back to a queue of waiting connections and retries, took all of it at once, hit the defect again and went down. By 23:55 UTC, the report says, restored servers were staying healthy briefly and then failing under traffic.

Figure 4. The recovery trap. A server returned alone absorbs the whole backlog and dies. Patching the defect and returning capacity in large batches breaks the loop.

The usual escape routes were closed too. Rollback, flighting and feature flags can undo yesterday's change, but this one had shipped in July and was baked into the build. Microsoft had to write a new update that returned the Windows code path to its previous known good state, then take groups of servers out of service, patch and restart them, and return them in big batches, often half a region or more at once, starting with regions entering business hours. Mail did not get lost in the meantime: sending servers received temporary 4xx errors and kept retrying, and Microsoft extended the delivery window to 48 hours so delayed messages would not bounce.

In an interview: when asked how a system recovers from overload, talk about bringing capacity back faster than demand can pile onto it. That means draining or rejecting the backlog, adding jitter and backoff to client retries, warming servers up with a ramp instead of full traffic, and returning capacity in large batches. Also say that every change should stay revertible for as long as it can hurt you, not just for the first few days.

What Microsoft says it has fixed

The preliminary report lists these repair items as complete:

  • An update that returns the connection handling code path to its previous known good state
  • A longer email delivery window, so delayed mail is not returned as undeliverable
  • Extra gateway capacity added proactively
  • More monitoring of traffic redirection patterns and connection handling health
  • Guardrails on traffic distribution to avoid cascading impact

The full list of longer-term changes is due in the final report.

How to use this in your next interview

Interviewers love real outages because they test whether you understand failure, not just the happy path. If you are asked to design an API gateway, an email service or a load balancer, you can now say exactly how the Microsoft 365 gateway failed and what you would build differently: separate pools for critical tenants, failover gated on spare capacity, load shedding at every server, bulkheads that traffic cannot cross, and a recovery plan that brings capacity back in batches.

Practise those answers on real problems:

Each one comes with an AI interviewer that pushes on exactly these failure cases.

Sources