In what is an unfortunate cost of doing business in the cloud era, an online hosting service “CodeSpaces.com” has ceased operations today after several days of online attacks. Ironically, only two days ago, CodeSpaces website claimed to provide “Rock Solid, Secure and Affordable” source code hosting, with a seven year history to back up those claims.
In what can only be called “One heck hack of a day” CodeSpaces released an ominous statement on Wednesday June 18th 2014, after suffering several days of DDOS attacks.
The statement read in part:
“An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address
Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.
Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had a already created a number backup logins to the panel and upon seeing us make the attempted recovery of the account he locked us down to a non-admin user and proceeded to randomly delete artefacts from the panel. We finally managed to get our panel access back but not before he has removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances.
In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.”
Like many services, the majority of CodeSpaces applications and data was itself hosted on a cloud provider, specifically Amazon AWS. This is not an indictment of Amazon, AWS or any other cloud service in particular. Rather, it should serve as a warning to anyone who believes there is little risk in moving applications and data into the cloud.
Like almost everything in the online world, the timeline of this disaster played out in record speed. As of 5am PDT on the 18th, the CodeSpaces twitter feed was directing people to “Read the latest updates at codespaces.com”, but just a few hours later, even the website was unreachable. It seems that some stale tweets may be all that remain of a company that just two days ago hosted thousands of development projects, tens of thousands of developers from over two hundred companies, including several Fortune 100 corporations.
The ramifications are staggering. How many thousands of lines of code have been lost forever? How many projects will suffer delays or perhaps even get cancelled as a result? Then, there is the more important question, “Has any code been compromised? Is my proprietary code now for sale on Silk Road 2.0?”
Certainly a post-mortem will occur which will undoubtedly point to a security breach within CodeSpaces procedures and potentially a lack of sufficient protection methods. However, it is doubtful that they are alone in their lack of safeguards.
This problem highlights a concern with any online, integrated infrastructure, where data protection mechanisms and long-term retention are all accessible using the same accounts and access methods.
While having separate data protection tools, media, procedures and interfaces can be cumbersome, it can also have benefits. A single point of access means a single point of vulnerability, as evidenced by this case.
In an online world, risks increase substantially with each reduction in complexity. In order to have both high resilience and availability, there are several requirements. It is necessary to have redundancy of three different items:
With these variables in mind, it is then important to create periodic, data protection points, which must then be copied between multiple accounts, locations and providers in order to enable the highest levels of reliability and availability. By removing any one of these variables, you have reduced your reliability, availability, resilience or all three.
Thus, the following may be thought of as a list of options, from least protected to more protected. Each of the Cloud Protection levels signifies the number of independent copies of data that exist.
There are several higher levels of protection that do not require additional data copies, but add encryption at one ore more locations. By adding encryption with private key management, three more levels of protection are available, which ultimately lead to “Cloud-9” (honestly, I intended only a small pun with this nomenclature).
In an online world, the risk of mistakes multiplies exponentially. If you accidentally leave your data-center unlocked, there is a good chance nobody would even find out that a door wasn’t locked. If they did, there is still a good chance that little, if any damage would occur.
In the online world, the risks are greater because the entire online population has constant access to every door. If anyone forgets to lock something, someone will find it. Once access is found, someone will exploit that weakness. Using the worlds most popular service provider with multiple data-centers is no guarantee that your service can’t be brought to its knees as CodeSpaces found out.
Perhaps the cloud motto should be “Trust nothing, lock your (crypto) keys and two copies are never enough.” Almost makes you nostalgic for tape backup doesn’t it?
