YHK - yet another hacker news clone

How would we learn about status in case of auth-adjacent incident?

Issue ID: MO1550821 Affected services: Exchange Online, Microsoft 365 suite, Microsoft Copilot (Microsoft 365), Microsoft Defender XDR, Microsoft Teams, SharePoint Online Status: Service degradation Issue type: Incident

User impact Users may be unable to connect to multiple Microsoft 365 services.

More info More info: Potentially impacted services include: - Exchange Online - Microsoft Teams - SharePoint Online - Microsoft Copilot 365 - Microsoft Defender XDR - Microsoft Planner - Microsoft To Do - Microsoft Purview: Multiple Purview services including Authorization, Auto Labelling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted.

Scope of impact This issue may affect users attempting to access the affected Microsoft 365 services.

Current status We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.

History of updates

- While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.

- We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.

- We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.

- We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.