Rackspace hosted Exchange suffered a devastating interruption starting December 2, 2022 and is still ongoing as of 12:37 AM December fourth. At first referred to as connection and login problems, the guidance was eventually updated to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be dealt with.
Clients on Buy Twitter Verification reported that Rackspace was not reacting to support e-mails.
This has actually been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or so. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Uncertain the number of companies that is, but it’s substantial.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t knowledgeable about the bounce for several hours.”
The main Rackspace status page used a running upgrade of the failure however the preliminary posts had no details other than there was an interruption and it was being investigated.
The very first authorities upgrade was on December 2nd at 2:49 AM:
“We are investigating a problem that is affecting our Hosted Exchange environments. More details will be posted as they appear.”
Thirteen minutes later Rackspace started calling it a “connectivity problem.”
“We are examining reports of connection concerns to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates explained the continuous problem as “connection and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination stage” of the outage, still trying to figure out what went wrong.
And they were still calling it “connection and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later on Rackspace described the scenario as a “substantial failure”and started providing their customers totally free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround till they comprehended the problem and might bring the system back online.
The official guidance stated:
“We experienced a considerable failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any further issues while we continue work to restore service. As we continue to work through the source of the problem, we have an alternate solution that will re-activate your capability to send and receive emails.
At no charge to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until additional notification.”
Rackspace Hosted Exchange Security Occurrence
It was not until almost 24 hours later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was experiencing a security occurrence.
The announcement further exposed that the Rackspace professionals had actually powered down and disconnected the Exchange environment.
“After more analysis, we have figured out that this is a security event.
The known effect is separated to a part of our Hosted Exchange platform. We are taking required actions to examine and secure our environments.”
Twelve hours later on that afternoon they upgraded the status page with more info that their security team and outdoors experts were still working on fixing the outage.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched details of the security event.
A security event typically includes a vulnerability and there are 2 extreme vulnerabilities currently in the wile that were covered in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack permits a hacker to check out and change information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assaulter is able to run destructive code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A confirmed remote opponent can perform SSRF attacks to escalate benefits and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mail box server, the aggressor can possibly access to other resources through lateral motion into Exchange and Active Directory environments.”
The Rackspace failure updates have actually not shown what the specific problem was, only that it was a security event.
The most existing status upgrade as of December 4th mentioned that the service is still down and clients are motivated to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in dealing with the occurrence. The accessibility of your service and security of your data is of high importance.
We have committed comprehensive internal resources and engaged world-class external expertise in our efforts to minimize unfavorable impacts to clients.”
It’s possible that the above noted vulnerabilities belong to the security occurrence impacting the Rackspace Hosted Exchange service.
There has actually been no announcement of whether consumer details has actually been compromised. This occasion is still continuous.
Included image by Best SMM Panel/Orn Rin