Microsoft outage: 'Millions and millions of dollars lost,’ partners say as Microsoft says ‘no evidence of malicious activity’
It had "a huge impact on business,” says Corey Kirkendoll, CEO of Microsoft partner 5K Technical Services. There has to be some sort of plan to be put into place for this. As one of my customers stated: ‘Unacceptable.’”
Microsoft’s global outage that impacted Teams, Outlook, Copilot 365 and more services led to Microsoft channel partners striving to help clients through the massive disruption.
Microsoft partners said the outage - with impacts ranging from police departments and governments to OpenAI and Anthropic - has cost customers millions of dollars.
“There were millions and millions of dollars lost yesterday [Friday 13th November] because companies could not work; they could not function,” said Bob Venero, CEO of Fort Lauderdale-based Future Tech Enterprise.
“What orders did they miss? What did they not see and get? What deals fell through because of this?” Venero said. “This is a wake-up call for the industry.”
Corey Kirkendoll, president and CEO of Plano, Texas-based Microsoft solution provider 5K Technical Services, said that the outage has been so disruptive to his customers that he’s fielding questions about moving to rival Google products.
“[The outage is] having a huge impact on business,” Kirkendoll said. “There has to be some sort of plan to be put into place for this. As one of my customers stated: ‘Unacceptable.’”
‘Millions of companies impacted’ in Microsoft outage; partners expect ‘a full explanation’
From police departments to local government offices, cities throughout the world reported email and communications disruptions.
OpenAI, for example, said it was finding issues with Microsoft connectors, with tools like ChatGPT not working “as expected.” According to multiple reports, the US House of Representatives said the outage impacted its users while its engineers are still working with Microsoft.
Future Tech Enterprise’s Venero said he has predicted for years that a major outage like this would have customers rethink their internal infrastructure related to communications, messaging and ERP.
“We have put too much into the hyperscalers and their infrastructure,” he said. “When something happens like this, it is catastrophic. It is not one company that is impacted. It is millions of companies impacted.”
Microsoft partner Phalanx Technology Group is frustrated with the ongoing issues.
“I expect a full explanation about what caused the outage and steps Microsoft will take to avoid this from happening again,” said Kelly Yeh, president of Chantilly, Va.-based Phalanx Technology Group.
“Right now, I will give the engineers space,” Yeh said. “I have been there when the unexpected happens and you’re in the middle of troubleshooting.”
Yeh said redundancy and failover measures should kick in for unexpected outages at prime business hours. “This is affecting many clients and partners, so I hope it gets resolved soon,” he said.
Microsoft outage not caused by ‘malicious activity’
Microsoft said on Friday that the outage was not caused by a cyberattack.
“At this time, we have no evidence of malicious activity associated with this incident.”
“We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defence-in-depth actions after mitigation is complete,” the company said.
The outage is still impacting Microsoft Copilot 365 as “users may be unable to load Copilot Chat or receive responses in the web and desktop apps, and Copilot experiences may be unable to access work data such as email, calendar, and files,” the company added.
Other popular Microsoft 365 services still affected include Microsoft Exchange Online, Microsoft 365 admin centre, Microsoft 365 Education, Microsoft Defender for Office 365, Microsoft Planner, Microsoft Purview and SharePoint Online.
Jason Wright, CEO of Avatar, a Houston-based MSP, said he has had no complaints from customers on the Microsoft 365 service issues.
“We’re moving from an era where businesses expected technology to always be available to an era where resilience is measured by how well they operate when it isn’t,” said Wright.
Wright said businesses are more resilient because they have communications parity in multi-cloud environments with multiple ways to communicate with customers.
The outage does show how critical email is to businesses, he added.
“People have been saying email has been dead for over a decade,” Wright said. “This shows that it is still the No. 1 source of communication.”
Microsoft needs on-premises outlook and 365, partner explains
Future Tech Enterprise, for its part, is urging Microsoft to bring to market an on-premises or air-gapped version of Microsoft 365 and Outlook.
Future Tech Enterprise has already started exploring on-premises email alternatives for customers but would prefer to have an on-premises version of Microsoft 365 or Outlook.
“I think there would be tremendous demand for an on-premises version of Microsoft 365 or Outlook,” Venero said.
“There are options out there from other companies, but they are not feature-rich. We don’t want to move off the Microsoft ecosystem. It is a great solution, but Microsoft needs to give us some flexibility so we can provide it on-premises,” the CEO said.
On-premises capabilities would reduce Microsoft infrastructure costs, Venero said. “Why not give customers the option to have an on-premises version, which would reduce their overhead because they wouldn’t need as many servers, infrastructure and support staff?” he said. “It reduces Microsoft’s risk.”
What Microsoft’s ‘focus’ is
Microsoft said it is seeing continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe.
The company said it is also seeing encouraging recovery signals in Asia-Pacific, although some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure.
“Our focus remains on stabilising recovered infrastructure, completing mitigation in the remaining impacted areas, and reducing delayed email queues and backlogs,” Microsoft said.
“We’re closely monitoring service consistency and validating recovery across different connection methods as we work toward full mitigation,” the company added.
Solution provider ‘grateful’ his business avoided the outage and customers ‘not blaming MSPs’
David Stinner, founder and president of MSSP US itek, Buffalo, N.Y., said he was grateful that his business was able to avoid the fallout from the cloud outage because his company uses Invarosoft’s managed endpoint pop-up software to communicate directly with customers even in the midst of the Microsoft 365 outage.
“It was critical that we had that platform that allowed us to tell our customers through screen pop-ups about the outage,” said Stinner. “That eliminated hundreds of phone calls into our help desk. It puts our customers’ minds at ease.
He said the Invarosoft product “is a critical part of our stack most MSPs don’t have.”
Even with the outage, US itek was able to get maintenance tickets through the Invarosoft application. “You can’t just rely on email for ticketing and communicating with customer,” he said.
Overall, customers were understanding given the widespread nature of the outage, said Stinner.
“A few years ago people didn’t understand the cloud,” he said. “Now they understand this is a Microsoft outage, and they were not blaming MSPs.”
Microsoft Teams and Microsoft 365 consumer still affected
Regarding continued issues with Teams, Microsoft said that, “Scenarios that depend on Exchange Online, including Calendar related failures with scheduling, invites, viewing, and meeting edits [are still impacted on Teams]. Creation of Live or Virtual events fail. Failures with Copilot responses for Outlook, calendar, or M365 files, read history, or adding to new chats. Navigating to App chats, adding new contacts or syncing contacts in settings fail. Standard channels are working fine but users may experience the inability to create or delete private or shared channels.”
On the Microsoft 365 consumer side, users may still be unable to access or perform several actions in Microsoft 365 services.
“Our recovery efforts to restore service health throughout the affected environment is progressing. In parallel, we’re continuing to pursue multiple workstreams to pinpoint the source of the issue and ensure the recovery of the affected environment. This issue impacts Outlook.com, OneDrive, OneNote, Microsoft To Do, Teams, and Microsoft 365 for the web although you may also see issues in other Microsoft 365 services,” Microsoft said.
Continue to follow CRN for more coverage of Microsoft's outage.
This article originally appeared on CRN UK sister websiteCRN US.