[conspire] Security breach @ multiple Federal agencies via SolarWinds Orion software

[Rick Moen]

In this posting, I'll be trying in real time to figure out the
substantive reality behind a current news story.  Example:
https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076

Headline is:
Feds Still Trying to Determine How Screwed They Are After Massive SolarWinds Hack
by Tom McKay

RM: There are recurring problems with IT press coverage of security
items, especially security breaches.  1. Where, as is frequently the
case, somebody messed up, the details go underreported because the
people who know don't want to talk about it.  2. IT reporters usually 
don't understand security very well, and tend to uncritically crib from
press releases.

  A cyberattack that began by targeting an IT firm used by numerous
  federal government agencies, Fortune 500 companies, and other high-value
  targets is shaping up to be a historic event.

  The U.S. government is still reeling after the detection of a massive
  foreign intrusion into federal computer systems at agencies including—at
  a minimum—the Department of Homeland Security, the Treasury, and the
  Commerce Department; [...]

  Those responsible built a backdoor into Orion, an IT management
  software produced by SolarWinds, possibly by breaking into Microsoft
  email accounts and other systems, according to the Wall Street Journal
  [link].  They then used it to contaminate software updates provided by
  the company with malware in March and June 2020. 

To unpack that:  Private US company SolarWinds, Inc. publishes
proprietary MS-Windows software for businesses to help manage their
networks, systems, and information technology infrastructure.  For
obvious reasons, any such software is itself security-sensitive and runs
with elevated privilege.  In Spring 2020, the Russian Federation Foreign
Intelligence Service ('SVR'), specifically its APT29 aka Cozy Bear team,
 managed to break into the crown jewels at SolarWinds, Inc., gaining
control of a software signing key for the production software chain,
which was then used to gain 'tokens' for other highly privileged roles
at SolarWinds, and among other thing insert remote-backdoor software
into a binary software library (SolarWinds.Orion.Core.BusinessLayer.dll) 
used in future releases of SolarWinds's Orion Platform software product.
So, the 'malicious' code in question then went out signed by
SolarWinds's release-code key, and so went out automatically to
customers as supposedly authentic code.

This root-level compromise of a piece of widely used commercial
off-the-shelf (COTS) software snagged _lots_ of victims.  Those who've
admitted getting suckered include:

o  NATO
o  US Treasury Dept.
o  US Commerce Dept. National Telecommunications & Information Administration
o  US Dept. of Homeland Security
o  EU Parliament
o  UK Health Service
o  UK Home Office
o  cybersecurity firm FireEye (!)
o  pharmaceutical and biopharmaceutical company AstraZeneca (probably)


How did SolarWinds, Inc. get H4X0Red?  Maybe, by being really
mind-bogglingly stupid?

https://www.msn.com/en-us/news/politics/notorious-hacker-fxmsp-sold-access-to-solarwinds-machines-report/ar-BB1bXZwj

  [...]
  Vinoth Kumar, a security researcher, told the outlet that he warned
  SolarWinds that their update server could have been accessed by "any
  attacker" with ease last year because the password was set to
  "solarwinds123." Kumar first notified the company of the issue on
  November 19, 2019 and the company responded three days later, according
  to emails he supplied to Newsweek.

  Kumar believes the vulnerability may have been present as far back as
  June 2018.
  [...]

Or maybe not?

  The recent breach, allegedly by Russian hackers, is also unlikely to
  be directly related to the password vulnerability since it took place
  months after the issue was remedied.

Doesn't seem reassuring, anyway.

SolarWinds, Inc. asks customers to un-fsck themselves as follows:

  SolarWinds asks customers currently using Orion Platform v2020.2 with
  no hotfix installed or 2020.2 HF 1 to upgrade to Orion Platform version
  2020.2.1 HF 2 as soon as possible to ensure the security of your
  environment.

https://www.solarwinds.com/securityadvisory/faq

If I were a customer, I'd want the answer to the question 'What
happened, guys, and why should I feel reasssured that it cannot ever
happen again?'  Is that addressed in their security advisory FAQ, you
ask?

  Why didn’t SolarWinds catch this vulnerability before it happened?

  This attack was very complex and sophisticated. The vulnerability was
  crafted to evade detection and only run when detection was unlikely.

Um, guys?

  How do you know the new build is secure?

  We have limited access rights to our build environment to only those
  necessary and added additional controls to limit access further. As an
  added precaution, we are using a new code signing certificate for our
  new builds.

Um, _guys_?  Why did this fail the first time?

  With these processes in place how was your code compromised?

  We are not aware that the SolarWinds code base was compromised.[...]

You're kidding.

  [...]Our initial investigations point to an issue in the supply chain
  resulting in a compromise of our product that inserted a vulnerability
  within its Orion monitoring products which, if present and activated,
  could potentially allow an attacker to compromise the server on which
  the Orion products run.

'An issue in the supply chain'?

Here's the thing:  If you do code-signing competently, you can no longer 
pass the buck to 'the supply chain', because any (hypothetical)
tampering downstream from your crown-jewels signing machine would result
in the modified software no longer validating as signed by the signing
key of record.

So, the logical inference is that the above is poppycock, that
SolarWinds's code-signing infrastruture, the crown jewels, was indeed
compromised.  And, by implication, SolarWinds, Inc. is either in denial
about this fact and is delusional, or is clumsily lying.  The latter 
interpretation would be a little more reassuring than the former, IMO.


Let's see what CSO Online says:
https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html

  SolarWinds stated that its customers included 425 of the US Fortune
  500, the top ten US telecommunications companies, the top five US
  accounting firms, all branches of the US Military, the Pentagon, the
  State Department, as well as hundreds of universities and colleges
  worldwide.

  The SolarWinds software supply chain attack also allowed hackers to
  access the network of US cybersecurity firm FireEye [...]

/me reads many more paragraphs.

Nope, no useful insights from CSO Online.

Brian Krebs (https://krebsonsecurity.com/) has started to cover the
story, but in fairness it's quite new.  (I expect he will have useful
things to say, and recommend his site.)

There's a subReddit to follow the story:
https://www.reddit.com/r/Solarwinds/



I'm going to have to close out this posting without any pretence of 
having reached grand conclusions:  Possibly, more will come out.
However, if I had to guess, based on available evidence, the root cause
will turn out to involve SolarWinds, Inc. security incompetence -- 
made worse by the shortage of transparency that is typical with
proprietary software companies.

You might wonder:  Could something similar happen with, for example, the 
Debian Project?  The simple answer is 'yes', but there is competent 
management of key-signing both at the ftp-master build machines and 
among the individual maintainers of Debian packages.  Basically, 
the all-volunteer Debian Project routinely does _way_ better than this 
major-name software company did.