[conspire] xz exploit and backdoor

[Rick Moen]

Quoting Ron / BCLUG (admin at bclug.ca):

> It cannot be overstated how devious this hack was!

Yes.  The technical details are impressive and twisty.
It was very stealthy in execution and well hidden in design -- really a
masterclass in how to do something covertly in a way well hidden against
scrutiny.

> It cannot be overstated how devastating this hack would have been.

Yes.  Almost all the world's Linux servers (the ones reachable in public
via SSH) getting coverly rooted, even for days, would have been
catastrophic.  And we really did come pretty close.

Also, it's still not completely clear what else the successful bits
allowed the unknown attackers into, or what other similar projects they
or others have or had.

> It cannot be overstated how lucky we all are that Adreas just happened
> to be testing at the right time, noticed delays of *under* ½ second on
> failed logins, and despite it being outside his area of expertise,
> decided to look into it further.

Andres has given an appearance on a security podcast, BTW.
https://risky.biz/RB743/  He's hilariously on-type.

> We'll have the discussion, then carry on as we are currently.

Probably.  But:  I know this is talking big, as I haven't exactly made
tracks on this larger project, but while thinking about next-gen system
software for my server, I've often thought I need do better at applying
the lessons of my Ops / InfoSec background.  And one of my recurring
frustrations with commodity distro packaged software (especially but not
limited to binary-package distros) is security-critical, public-facing
software being overfeatured by selection, by configuration, and by
choice of compilation options.

Let me back into what I'm thinking about.

Some years ago at $FIRM, an important part of my job was to be one of
two senior sysadmins for a semi-autonomous division functioning as a
"merchant bank".  In the USA tech field, that is a term of art meaning 
"credit card transaction processor for e-commerce".  Other companies'
public-facing online "stores" could back-end into $FIRM's "merchant
bank" to accept and process the card charges.

All merchant banks must periodically (every 6 mos., if memory serves)
pass "PCI testing", the Payment Card Industry assocation's Payment Card
Industry Data Security Standard (PCIDSS) security-quality testing.  And
I was the lucky sod who had to shepherd our farm of CentOS-based servers
through that periodic process.

PCIDSS testing was, in a way, a little lame.  A chosen PCIDSS testing
vendor would, twice a year, port-scan your IP netblocks with some
secret-sauce perl script, making guesses based on fingerprinting of what
public-facing software comprised your public attack surface, and then
you as the customer (and I as the fix-it guy at the customer) would get
a report saying "Hey, our probe claims you're running Foo daemon version
XX.YY.Z rev. A, so kindly prove to us that your implementation isn't
vulnerable to the following CVEs...."  And each of those would be a CVE
that _might_ apply to Foo XX.YY.Z rev. A.  (Likewise for other
software.)  My job would be to prove to the vendor that our site
_lacked_ that possible vulnerability, either because we'd upgraded to
CentOS update this-or-that, or because we'd specifically disabled that
functionality entirely in /etc/foo/*.conf .  

Sometimes, the vendor's claim would be one of many bizarre types of
false positive.  Then, my job was to explain why their script had been a
prat and was mistaken.  Either way, my explanations (rarely, after
rolling out some upgrades or conffile mods) would be sanity-tested by 
some functionary, and (at the end) accepted as a "pass" for 6 months.

It worked well enough, and we never had security incidents -- to our
knowledge, at least.

Getting back to distro software choice and packaging:  The "every
possible use-case must be supported out of the box" syndrome is
dispiritingly ubiquitous.  Some stuff is compiled into a daemon binary 
even though almost nobody will use it, just because someone might.
Why is Apache httpd almost always the default, rather than Lighttpd,
nginx, sthttpd, or Hiawatha?  Because its big dumb software that does
everything, that everyone's used to.

$FIRM ran Apache httpd, but many PCIDSS complaints got finessed by
sending them our conffiles where various buggy features were
specifically disabled.

Network Time Foundation's NTP Project software, used by damned near all
server-oriented distros by default, is likewise big dumb software that
does everything, that everyone's used to.  OpenBSD Foundation's OpenNTPd
was founded with the specific aim of avoiding NTP Project's featuritis, 
and has succeeded (with only minor cavils).  Red hat's Chrony is a
similar story.

I'm sure you get my point:  You get a smaller attack surface by choosing
security-relevant software with minimal unneeded servers coded in, and if
possible further cut back on unwanted functions in the runtime config.

And, last, there are times you want to rebuild a distro package from
source to use different compile options.  For daemons, this can be a
reasonable local admin burden, depending.  For example, imagine I were 
looking at the (newest) Debian Portable OpenSSH package's dependencies,
and said "Eh?  Needs liblzma.so.5?  I don't need that.  I don't _want_
that."  I'd check, and find that the package got compiled to require it,
in order to be able to report service-readiness to libsystemd -- which 
I would not have any interest in, therefore compiling a local OpenSSH 
package with that compile-time option disabled would do the trick.

But Portable OpenSSH isn't the only choice, either.  There are respected
SSHDs that are _much_ smaller/lighter with a smaller attack surface,
like Dropbear and wolfSSH.  "But they're less well known!"  Do I care?
Hell no.

Anyway, it seems to me that substantial improvement can be made on a
local system through active system administration, picking wiser choices
of software packages, paring down configurations, and in some cases
compiling locally to reduce runtime dependencies.

All of which is only a _little_ helpful against insider threats such as
"Jia Tan", whoever that was.  But OTOH shying away from big dumb
software and featuritis can help defang even the Jia Tans.

> There's been some interesting analysis of the timestamps of JiaTan's git
> commits showing odd patterns.
> 
> Jia claims to be from California (GMT -8) but timestamps tended to be
> GMT +8 - same as Singapore.
> 
> Many commits were GMT +2 or +3 (DST dependent), pointing to Eastern Europe.

APT29 aka Cozy Bear is a popular guess.
https://en.wikipedia.org/wiki/Cozy_Bear

The list of possiblities is very broad, though.

-- 
Cheers,             "Are you sure it’s that simple?  After all my time here, 
Rick Moen           I’ve yet to see any problem, however complicated, which 
rick at linuxmafia.com when you looked at it the right way, didn’t become still 
McQ! (4x80)         more complicated."     -- Poul Anderson, in "Call Me Joe"