Microsoft: July patch caused November email meltdown

Original: Microsoft blames monster email outage on software defect introduced in July fix

Why This Matters

Exchange Online downtime at this scale exposes fragility in Microsoft 365's global infrastructure.

Microsoft confirmed a software defect introduced in a July 2026 patch triggered last week's Exchange Online outage, leaving over 80% of gateway servers unresponsive at peak failure on Nov. 15.

Microsoft's preliminary post-incident report, published November 15, pins last week's massive Microsoft 365 email outage on a code change quietly deployed in July 2026. The fix passed load testing at the time and was rolled out without apparent issue — but the testing had a blind spot.

Microsoft's own report admits: "Our load testing tended to focus on weaker hardware with fewer cores." The defect triggered lock contention inside an internal Windows process, exhausting CPU and memory — a problem that gets worse, not better, on newer, more powerful hardware.

The actual failure came from two simultaneous events. Teams failed over to the primary gateway supporting Exchange Online, spiking load. Simultaneously, nearly 14% of gateway server capacity was offline for maintenance. Neither event alone would have caused trouble. Together, they pushed servers into the hidden defect.

What followed was a textbook cascading failure. Servers fell over; traffic rerouted to the next healthy endpoint, which then fell over too. Within two hours, the defect had spread across Microsoft's entire fleet. At the worst point, more than 80% of gateway servers were unresponsive. Rolling back wasn't an option — returning servers to service just meant they were immediately overwhelmed again, forcing engineers to patch the patch rather than revert.

Source

theregister.com — Read original →