41 points Bender 2 hours ago 46 comments
ncr100 1 hour ago | parent
Is this commonly acceptable? I'd imagine it's a costly trade-off?
bayindirh 56 minutes ago | parent
But if low is 64 and high is a whole cluster, now that's a different kind of maths.
Edit: The article states the following:
> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."
So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.
Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?
(Man, I felt old while writing this.)
lokar 31 minutes ago | parent
feverzsj 1 hour ago | parent
elcritch 1 hour ago | parent
rs999gti 49 minutes ago | parent
You mean you and me? You realize pensions, institutions, and individual investors are shareholders as well?
elcritch 20 minutes ago | parent
The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.
Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.
In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.
triceratops 10 minutes ago | parent
dgrin91 54 minutes ago | parent
I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.
The response clearly could have been stronger though.
thesuitonym 42 minutes ago | parent
zdragnar 40 minutes ago | parent
preommr 53 minutes ago | parent
Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)
We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.
hmokiguess 57 minutes ago | parent
What does this mean? I didn't understand
abrichr 54 minutes ago | parent
bayindirh 54 minutes ago | parent
In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.
Molitor5901 38 minutes ago | parent
voidfunc 23 minutes ago | parent
Surprised so many people are struggling with this...
Having been on incidents like this before though, yea, that aint happening.
27183 18 minutes ago | parent
Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".
bayindirh 17 minutes ago | parent
Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.
Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.
Glad I'm not there anymore.
giancarlostoro 9 minutes ago | parent
https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...
How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.
sandworm101 53 minutes ago | parent
I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.
bayindirh 45 minutes ago | parent
Here, I converted it to corporate speak for you.
thesuitonym 40 minutes ago | parent
betaby 36 minutes ago | parent
nashashmi 31 minutes ago | parent
nashashmi 33 minutes ago | parent
stackskipton 35 minutes ago | parent
Microsoft does greenfield new stuff on Linux mostly.
vessenes 44 minutes ago | parent
toss1 41 minutes ago | parent
Indeed they do deserve a long hard look.
Toss1's conjecture:
We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.
(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])
[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...
nashashmi 37 minutes ago | parent
Better summary: https://www.neowin.net/news/microsoft-finally-explains-why-o...
ZiiS 33 minutes ago | parent
- our testing didn't cover it (again, this is expected occasionally).
- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).
- it took 15 hours to deploy the fix (not good, but at this scale maybe).
Feels like they are completely missing the point of where the problem is and how to fix it?
lokar 29 minutes ago | parent
bayindirh 13 minutes ago | parent
So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
lokar 5 minutes ago | parent
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
bayindirh 3 minutes ago | parent
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
lokar 42 seconds ago | parent
spprashant 29 minutes ago | parent
But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.
When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.
dgellow 5 minutes ago | parent
That’s Silicon Valley fantasy, it’s not even close to reality
wduquette 25 minutes ago | parent
mxuribe 23 minutes ago | parent
dgellow 8 minutes ago | parent
colonCapitalDee 17 minutes ago | parent
ChrisArchitect 9 minutes ago | parent
mike_hearn 7 minutes ago | parent
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.
Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.