[conspire] xz exploit and backdoor

[Rick Moen]

A huge story hit the Linux / open source / Information Security
(InfoSec) world this past Friday, and it's both shocking and a sign of
some basic problems we need to fix, or next time we might not be so
lucky.

An extremely devious team (unknown) started work several years ago on a
scheme to security-subvert essentially all public-facing Linux servers,
and very nearly succeeded.  The scheme got blown up by a Microsoft
database engineer, puzzled about why his PostgreSQL sessions over SSH 
transport were consuming too many CPU cycles, generating unexpected
memory-metrics errors, and slowing down logins.

Profiling the SSHD process (on his Debian "sid" = unstable branch
PostgreSQL server), the developer, Andres Freund of San Francisco,
traced the performance bottleneck to library liblzma (part of the
xz-tools compression toolkit), which gets linked into the Portable
OpenSSH daemon on many Linux distributions in order to support
daemon-readiness notification to libsystemd, xz compression being needed
at that point to compress system journal files.

Freund kept poking, and found that the liblzma code included a secret
backdoor usable on any server where it's a dynamic dependency for
/usr/sbin/sshd. 

And so, Freund told us-all about that, last Friday:
https://www.openwall.com/lists/oss-security/2024/03/29/4

The way this was implemented was extremely twisty.  The bad guys made
very few mistakes, were playing a very long game, and very nearly won.
In fact, the full backdoored sshd was included in recent Debian
testing/unstable, Fedora beta & Rawhide, openSUSE Tumbleweed, and Kali
Linux.  The circle of effects keeps widening:  Some of the attacker's
worrisome code made it into Homebrew for MacOS, and into MS-Windows 11, 
and the hunt continues.

It's a very complex (and still unfolding) story, told in a number of 
good places:

https://lwn.net/SubscriberLink/967866/5795ab3964643db6/
https://www.schneier.com/blog/archives/2024/04/xz-utils-backdoor.html
https://www.wired.com/story/xz-backdoor-everything-you-need-to-know/
https://research.swtch.com/xz-timeline
https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
https://infosec.exchange/@fr0gger/112189232773640259
https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27#faq-on-the-xz-utils-backdoor
https://tukaani.org/xz-backdoor/

There's some evidence the bad guys had to rush their plans because
Systemd has a change pending to reduce libsystemd's dependencies on
general security grounds, which would break the backdoor completely.
Also, Portable OpenSSH had plans for a method to support
daemon-readiness notification without need for linking in an external
library.

It's important to realise that the _particular_ chain of embuggerment
these attackers used (not fully detailed here, by half) didn't matter
much:  Many others could have served, and we-all need to have a long
discussion about software complexity, excessive dependencies, how to
deal with insider threats, and developer burnout.

Oh, right, that last bit was crucial.  Lasse Collin, the innocent sole
maintainer of xz-utils, was targeted in a brilliant _social_ attack as
the first step of getting into Portable OpenSSH on Linux.  If you look
into nothing else on this story, watch this video about how Collin was
manipulated into giving developer access to pseudonymous developer "Jia
Tan" (username JiaT75), here:
https://www.youtube.com/watch?v=0pT-dWpmwhA

_And_, it's an open question what other backdooring might have quietly
succeeded already.

Interesting times.

-- 
Cheers,             "Are you sure it’s that simple?  After all my time here, 
Rick Moen           I’ve yet to see any problem, however complicated, which 
rick at linuxmafia.com when you looked at it the right way, didn’t become still 
McQ! (4x80)         more complicated."     -- Poul Anderson, in "Call Me Joe"