.png&w=800&c=0&s=1)
The SolarWinds attack has gained much coverage, what with 425 of the US Fortune 500, the top ten US telecommunications companies, the top five US accounting firms, all branches of the US Military, the Pentagon and the State Department, as well as hundreds of universities and colleges worldwide being part of their customer base.
In a recent media briefing held by SolarWinds, Tim Brown, Vice President, Security and Chief Information Security Officer, and Brandon Shopp, Vice President of Product Strategy, Network, shares their perspective on what happened and measures the company is undertaking to update and deploy their software safely.
iTNews Asia: How did the attack happen, and what exactly are SUNBURST and SUPERNOVA?
Shopp: SUNBURST was effectively a stage 1 attack that gave initial access to the bad actors to an environment if a customer had installed that version of the product. It was actually code that was shipped with the product, injected into the application as part of our build processes. SUNBURST is only known to affect three versions of the Orion family products.
There's some other terms that you may have heard come out, either from us or from other government agencies or law enforcement agencies. SUNSPOT is the tool that was used to inject the SUNBURST code into the Orion product.
So if you kind of think about it as an umbrella, SUNBURST was the overarching umbrella from an event perspective. Then there were a set of elements as part of that supply chain attack – SUNSPOT being one of them, and TEARDROP, RAINDROP, SUNSHUTTLE as well as GOLDMAX that are stage 2 elements.
In the way that would work is – a customer installs an impacted version of the product into their environment. That does not by default mean that they have been compromised. It just means that they are running an impacted version. The code itself within Orion would wait anywhere from 11 to 14 days, before it would activate and run through a set of self-checks before attempting to establish contact to an outside domain.
If that was done successfully, then it would reach out to the command-and-control server and then basically await the next set of instructions. So this doesn’t mean that the environment was compromised. It just means that the bad actors then had a list of organisations or installations that they can choose in which to take the next stage with to then go and potentially do lateral movements or other activities within that end-users environment.
Brown: Just one thing about the attacker – a lot of the testimonies from Microsoft, FireEye, CrowdStrike, and others – occurred in the last few weeks to the US government. They and the US government are attributing the attack to Russia right now. Simply because of the methods that were utilised and the sophistication of the attacker.
The attack didn't come in as extremely disruptive. It was after a single mission, and that mission was to be able to compromise Orion, have us ship a piece of code in Orion, and then target a very small subset of customers that could be affected.
The threat actor was not looking to do random harm to companies, but what it did was provide a back door to the operating system that was running Orion. Then allowing the threat actors to try to move laterally within that environment to go from the operating system to other places.
Shopp: Another term that you may have heard is SUPERNOVA. We don't believe that SUPERNOVA and SUNBURST are directly related. We believe that they are two different incidents.
SUPERNOVA was not something that was injected into Orion, and it was not something that we shipped. It was a bad actor that ultimately exploited a vulnerability that happened to be in the product. In this part of the stage 2 attack was ultimately the SUPERNOVA malware, the attack itself.
In order for the bad actor to be able to exploit that vulnerability, they would have had to have already compromised that environment in some form or fashion. So either it was an internal bad actor or again they compromised through another means into that end users environment.
The bad actors would then scan or look into that environment for other vulnerabilities that they could leverage and exploit. Orion did have that vulnerability that we have since addressed in late December with some patches that we have put out.
One of the key markers for somebody to see if they happen to be compromised was to look for a DLL that was not signed because the bad actor effectively swapped out a good file that was signed by us with one that was unsigned by them. We don't believe at this point that this is the same set of bad actors, and we don't believe that these two incidents or these two events are related. They just happen to come out and become known roughly at about the same time period.
iTNews Asia: How did SolarWinds find out about the attack and what was done to stop it?
Brown: On December 12, we were informed by FireEye that we had shipped out the tainted code. Did we know about it before? Absolutely not. We did not know about it until December 12 when we were informed.
Since it occurred, we have engaged a number of external third parties – the FBI, to be able to start an investigation of what happened, the Cybersecurity & Infrastructure Security Agency (CISA) for the US government, and a number of other international agencies that we have engaged in.
We could see which versions were tainted. We understood that, but we knew we had to do a major investigation within the company. With that we essentially started a threat hunt internally – an investigation of if the threat actor is still here.
We engaged two different entities to be able to do that investigation – CrowdStrike and KPMG. CrowdStrike is focusing on the macro investigation while KPMG focused on the micro view.
Basically all the servers and the workstations that are utilised by SolarWinds were put up with the CrowdStrike Falcon set of tools – a number of threat hunting tools on each one of our boxes.
What that allowed the threat researchers to do was really look over the environment, get full visibility, search the environment, look for any signs of lateral movement or that the threat actors were still there.
Meanwhile KPMG is investigating our build environments. They were investigating anything that needed forensic analysis. They investigated the build environments for not only Orion, but the other environments around the company.
They also investigated the source code, and the source code control system. We don't see any signs that the source code control system was tampered with at all. So the insertion point was not tainting the source code, but it was tainting the build process. It is important to note that because it is an important component of how the supply chain works.
With CrowdStrike and KPMG's help we actually found that code called SUNSPOT, and we then made that SUNSPOT code public. The reason why we made it public is that that model could be utilised for anybody building any type of software. So it was important for the whole industry to understand that this was a possibility.
Shopp: One of the things that we've done since the incident was revoke the digital code signing certificate used with those impacted builds. The revocation occurred last Monday (8 March).
With that, we released a couple new builds for our customers that are signed with our brand new digital code signing certificate. This is what we have been asking and driving our customers to upgrade to as the safe spot for them to ultimately be at.
We have had thousands of customers upgrade to these (new Orion) versions, and we have been watching very closely for any sign of malicious activity. Through thousands of upgrades to the latest versions, we have seen a couple of false positives, but nothing from a true malicious intent on any of these products.
- Tim Brown, Vice President, Security and Chief Information Security Officer
Shopp: We are also in the process of migrating to a next-gen build environment, doing a dual build verification process. When we grab the source files from the source control and building those through the build pipeline, we get the final set of installers at the end of that that are signed. So we are taking those, installing them in our environment, and decompiling those back to source.
Why we're doing that is so that we can get down to each of those decompiled source files, get down to the hashes associated with those and compare those against the hashes of the same files in the source control. This will allow us to ultimately ensure that there was no tampering or insertion throughout that build pipeline because what went in is identical to what came out at the end.
We have gotten to the point where we're not just inspecting our own source files, but those from third party libraries as well. We have worked and spoken to these vendors, and they have provided us with the applicable hashes for those packages that we happen to use. So we're in the latest builds, effectively getting 100% coverage to where we are able to confirm that what is in the final installed built product is identical to what is in our source code repository.
iTnews Asia: What sort of help has been extended to your customers?
Shopp: First off, we notified all customers that we believed to have downloaded the affected versions within about twenty-four hours after we were first notified by FireEye.
We were being communicative and transparent in letting them know about what we know at this point, and what we're doing and what we believe were products that were impacted and which ones were not.
We have continued those communications over the past three months on our security advisory page on our website as well as our FAQ. We have continued to refresh content as well as through our Secure by Design webinar series that we have been doing to make sure that we're sharing information with our partners, customers and the broader IT Community as a whole.
Brown: It was very important that when people reinstall or upgrade, that they do so in a secure fashion. So the Orion secure deployment guide pulls a bunch of information together on one document on how to appropriately configure, install and how to make sure that your installation itself is as secure as possible.
Shopp: We provided updated guidance on working with CISA as well our third parties. So for customers that needed to patch or upgrade their Orion to the latest version, we created the Orion Assistance Program or OAP. It is for our active maintenance customers that we are providing at no charge. Effectively what we are doing is leveraging on third-party partners to aid in in that upgrade process.
