[sf-lug] (forw) Re: (forw) Re: Something new on Distrowatch and Ubuntu variants.
Rick Moen
rick at linuxmafia.com
Mon Apr 26 22:58:01 PDT 2021
Quoting Al (awsflug at sunnyside.com):
> Now not to be totally pessimistic and negative, the big sites like
> debian and ubuntu will have (or so I believe) checksums that are
> actually correct. I checked some once. Mostly I just download
> from big established well run websites and trust, but there's that
> Mint case... I hadn't even heard of that.
Here ya go:
I'm going to re-post details from one of the several _previous_ times
I've tried to advise Bobbie she ought to start vetting developer
keysignatures (where available) on the ISOs she hands out -- this
occasion having been a bit over four years ago -- and some of the
resulting thread.
Date: Sun, 21 Feb 2016 16:26:34 -0800
>From rick at linuxmafia.com Sun Feb 21 16: 6:34 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: How to check distro checksums and signatures (was: Linux Mint
iso
files hacked.)
Organization: If you lived here, you'd be $HOME already.
Quoting Bobbie Sellers (bliss-sf4ever at dslextreme.com):
> Linux Mint iso files hacked!
> IF you downloaded a Linux Mint iso on February 20, 2016
> It may be compromised with a backdoor.
Bobbie, first, thank you for passing along that news item.
At the same time, that article's contents isn't within a country mile of
the most important lesson to learn, but I can help:
What you should ask yourself is: What about _any_ downloaded program
(and distribution isos are collections of many programs) makes it worth
trusting? How do you have reasonable assurance that a program you
retrieve isn't a trojaned fake someone inserted into the download
process?
Linux distros use checksums and cryptographic signatures as a safeguard
-- and you should make sure you check those.
When you use package managers to upgrade/install distro-produced
packages, the process happens transparently to you. Your running distro
has a crypto 'keyring' that it uses to check signatures on packages, and
the package tools also verify checksums to make sure the downloads are
complete and intact.
By contrast, when you manually download files (such as isos, the burden
is on _you_ to do that checking manually.
Here's the Linux Mint download page:
http://blog.linuxmint.com/?p=2947
Page starts out with text introducing Linux Mint 17.3 'Rosa' Cinnamon
Edition, and goes on to a 'Direct download' section (alphabetical list
by country of download mirror sites). _Below_ that long list of
countries is this bit:
Signatures (to verify your downloaded ISO):
MD5SUM 64-bit: e71a2aad8b58605e906dbea444dc4983
MD5SUM 32-bit: 6e7f7e03500747c6c3bfece2c9c8394f
Signed Sha256sum signatures [link]
This stuff (above) is absolutely _vital_. You should always look for
it, and _not_ just go straight for the download link. Your first
thought at the time of looking-for-downloads should be 'Where are the
checksums, and are they signed?'
Checksums currently come in three flavours. From weakest to strongest
(degree of cryptographic assurance of correctness) types, they are:
o md5sum
o sha1sum
o sha256sum
A checksum, aka a hash, is a number derived from a file, and much
smaller than that file, that you can use to verify data integrity by
recalculating the checksum of the file you received and making sure it's
the same as what the download site claims it should be. If it's a
match, then you have pretty strong proof your file download wasn't
truncated or corrupted in transit.
The utilities to do this task in Linux are named (yay!) md5sum, sha1sum,
and sha256sum. Check your distro; it'll have them already.
Let's say you download linuxmint-17.3-cinnamon-64bit.iso. OK, now you
immediately do:
$ md5sum linuxmint-17.3-cinnamon-64bit.iso
If your download was complete & uncorrupted, /usr/bin/md5sum will
respond:
e71a2aad8b58605e906dbea444dc4983
(The checksum tools also have a '-c' option to do the comparison for you
more automatically.)
Verifying the checksum is step #1 of 2. The second step is checking the
gpg[1] signature of the checksum -- because, unfortunately, while
recalculating the checksum verifies completeness, it doesn't verify
authenticity, because the same Web site intruder who modifies Linux
Mint's page to post links to fake isos might also alter the page to
specify fake checksums.
So, you use /usr/bin/gpg to verify signature of the posted checksum. I
was going to show you how to do that with Linux Mint, but
unfortunately it seems they muffed this information: As you'll see
in the quoted 'Signatures' text (above), they posted md5sums but not
_signatures_ of those md5sums. They published signatures of sha256sums,
but not the sha256sums themselves.
This post is already long enough, so I won't invent an example of gpg
checking. Perhaps some other time, or someone else might speak up to
show how one does it. Example tutorial:
https://help.ubuntu.com/community/VerifyIsoHowto
You should be asking yourself, 'OK, if checksums check competeness but
not authenticity, and signing is used to check authenticitity, what
checks that the signatures are authentic? Can't the same intruder who
modifies a distro download page to post links to fake isos and fake
checksums also modify it to show fake gpg signatures?'
Yes, indeed, and this is where signatures being known or not is useful.
If the key used to sign a distro's isos suddenly changes, many people
who've known and relied on that key will quickly notice and raise the
alarm. Also, previously unseen gpg keys can be checked by using gpg to
note who has vouched for them (signing those keys with their own keys).
In this case, the alarm got raised quickly because Linux Mint's md5sums
aren't published in just one place but in many, and downloaders checking
the fake isos (in many cases) got unmatching checksums.
How did the compromise occur? It occurred because the affected server
runs WordPress, which IMO has terrible security, just like many other
popular PHP Web applications. (PHP, IMO, is a security menace
generally.) Clem Lefebvre of Linux Mint says: 'We found an uploaded
php backdoor in the theme directory of a wordpress installation, which
was 1 day old and had no plugins running. The theme was new but most
importantly I think we had lax file permissions on this.'
The intruders were then able to run OS shell as user 'www-data', the
user that runs Apache httpd. Note: If you run a Web server, it is very
important as a fallback security measure to ensure that user www-data
has restricted privileges. Even then, hostile processes running as that
user could do serious mischief, and are a serious matter.
I just posted this to Clem's blog page http://blog.linuxmint.com/?p=2994
(awaiting moderation):
Clem, your point is good one that duplication and the community was an
effective cross-check and instrumental in spotting the compromise
quickly. But (and you knew there was a ‘but’, right?) the people on
this
thread suggesting improved gpg-signing of checksums also have a valid
point.
You said ‘You can find them at
http://ftp.heanet.ie/pub/linuxmint.com/stable/17.3/ also along with
signed sha256sums’ and ‘we’ll probably default to showing sha256 for
upcoming releases’ — which is good news. However, please note that
primary download pages such as http://blog.linuxmint.com/?p=2947 have
for a long time (and still) listed md5sums and gpg signatures of
sha256sums, but not included gpg signatures of md5sums, and not
included
sha256sums. So, unless a member of the public thinks to also look on
http://ftp.heanet.ie, he/she could not easily check gpg signatures at
all.
I would like to politely suggest that you good folks take a careful
look
at the published means of verifying authenticity, and make sure
everything works even for half-clued outsiders, and that this include
care to make sure signing keys are publicised and able to be vetted
using the gpg chain of trust.
Thank you for Linux Mint, and for your good work.
Best Regards,
Rick Moen
rick at linuxmafia.com
Well, OK, since this post is already windy, I'll chance a bit more
breeze:
Let's say you somehow stumbled across
http://ftp.heanet.ie/pub/linuxmint.com/stable/17.3/ that has _both_ the
sha2sums and the gpg signature for those sums. You fetch
sha256sum.txt.gpg and sha256sum.txt to your local disk. Then:
$ gpg --verify sha256sum.txt.gpg sha256sum.txt
gpg: Signature made Wed 06 Jan 2016 08:06:20 AM PST using DSA key ID
0FF405B2
gpg: Can't check signature: public key not found
$
This is not surprising, because your system (or at least mine) hasn't
previously encountered the gpg with 'DSA key ID 0FF405B2'. (That is
a hash of the Clem Lefebre's signing key by the way, a hash.)
So, you ask the public keyservers about that key:
$ gpg --search-keys 0FF405B2
gpg: searching for "0FF405B2" from hkp server pool.sks-keyservers.net
(1) Clement Lefebvre (Linux Mint Package Repository v1)
<root at linuxmint.co
1024 bit DSA key E1A38B8F144675D060EA666F3EE67F3D0FF405B2,
created: 2009-04-29
Keys 1-1 of 1 for "0FF405B2". Enter number(s), N)ext, or Q)uit > 1
gpg: requesting key 0FF405B2 from hkp server pool.sks-keyservers.net
gpg: key 0FF405B2: public key "Clement Lefebvre (Linux Mint Package
Repository v1) <root at linuxmint.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg: imported: 1
$
And now, to no great surprise, the signature checks against the public
key you just fetched.
$ gpg --verify sha256sum.txt.gpg sha256sum.txt
gpg: Signature made Wed 06 Jan 2016 08:06:20 AM PST using DSA key ID
0FF405B2
gpg: Good signature from "Clement Lefebvre (Linux Mint Package
Repository v1) <root at linuxmint.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the
owner.
Primary key fingerprint: E1A3 8B8F 1446 75D0 60EA 666F 3EE6 7F3D 0FF4
05B2
$
In the 'WARNING' text, gpg is properly paranoid, pointing out that you
have no special reason to think the public keyserver's entry is actually
Clem's key: gpg couldn't find any signature _of_ Clem's key to furnish
an additional reason to trust it. However, you can reasonably expect
that if the public keyservers had a fake key for Linux Mint and Clem
Lefebre, people would notice quickly.
If checking signatures seems too complex, _at least_ verify (recalculate
after download) the checksums -- which is easy. In the current Linux
Mint case, for example, verifying checksums was sufficient to catch the
forgeries.
> Full story and comments(read them) at:
> <http://betanews.com/2016/02/21/linux-mint-hacked-iso-image-compromised/>
You know, Bobbie, intending no criticism of you personally, but the
above news story provides nothing whatsoever about how to detect
forgeries. It just says 'beware of this one forgery'.
Above I've attempted to fix that grievous omission.
Please, folks, don't just download executables (including isos) and
trust them. Stop that, please. Use checksums. _Use signatures._
> I downloaded the iso images of Mint that I have at the meetings long
> before this happened, if anyone has any concern.
Yes but also: Use checksums. Use signatures.
[1] 'gpg' is the open source reimplementation of PGP, Phil Zimmermann's
famous Pretty Good Privacy program. Be aware that gpg has an
infamously confusing and opaque command-line interface. Don't hesitate
to Web-search for tutorials such as the Ubuntu one I cited above.
Date: Sun, 21 Feb 2016 16:52:03 -0800
>From rick at linuxmafia.com Sun Feb 21 16: 2:03 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: Re: [sf-lug] Linux Mint iso files hacked.
Organization: If you lived here, you'd be $HOME already.
Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
> One of the things that has always caused me to not consider Linux Mint
> to
> be a "serious" Linux distribution, is they *never* established a
> trust path to their ISOs! Sure, they'd put a hash of the ISO on their
> web site. But not even SSL for the web site, and no signature for the
> ISO file.
1. SSL for the Web site wouldn't have adressed this current threat
model,
especially given that the actual downloads are on a large number of
public mirror sites. (HTTPS is a good idea on its own merits, but would
not have prevented this attack.)
2. They do post signatures for the checksums of their ISOs, but they've
been sloppy about it. And although Clem's signing key for those
signatures is in the public gpg keyservers, it isn't well attested.
(Near as I can tell via a quick check, there's no chain of trust to
follow. I could be missing something.)
I hope the recent WordPress PHP compromise is a sufficient wake-up call
to motivate them to address point #2 above.
As you'll see in my separate post, I gave Clem a polite kick about that.
> Even where they post about the compromise:
> http://blog.linuxmint.com/?p=2994
> The give no trust path to the hashes (and egad, still using md5sum, at
> that).
The relative weakness of md5 hashing is the tiniest part of this
problem, and trivia by comparison. When your security is so tight that
it's worth your time to worry about exploitable md5 hash collisions,
you've done well indeed. A much bigger problem, IMO, is that they
omitted the stronger sha256 hashes from the main download page --
suggesting a slapdash attitude to verification and integrity-checking
generally.
> And ... only 1024D?
That might be a matter of when the key originated. Until fairly
recently[1], DSA public key encryption was limited to 1024 bits.
By comparison, RSA key length was never limited, so one has always been
able to generate much stronger RSA keys than DSA keys -- which IMO is
one of many reasons to prefer RSA. (About all DSA ever had for it was
patent avoidance, and that went away when the RSA patent expired in
2000.
> And is there trust path to the signing key?
Probably not. But they've done a terrible job of even providing public
information about how to get the checksum matching the signature, and
the public key matching the signature. Seems like the biggest problem,
to me.
[1] NIST defined DSA in fips 186-2 with a 1024-bit limit in the
standard, but that has been superceded twice, the current revision being
fips 186-4, http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf.
I'm not sure when that change occurred, but probably not too many years
back. FWIW, current revision fips 186-4 is dated July 2013.
Date: Mon, 22 Feb 2016 00:12:51 -0800
>From rick at linuxmafia.com Mon Feb 22 00: 2:51 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: Re: [sf-lug] Linux Mint iso files hacked.
Organization: If you lived here, you'd be $HOME already.
Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
> Yes, you make good points about the relative (un)importantce of the
> various pieces present - and missing. Perhaps I was quite meaning to
> imply, but didn't explicitly state, more generally their relatively
> lackadaisical (in)actions regarding assurances of the integrity of
> ISO downloads. [...]
Just a brief note to say thanks and that we're also in agreement.
It's vexing that Linux Mint has a record of half-assedness about this
matter. Let's hope that they have been shocked into doing the missing
30% of the job -- because they actually have most of it. The only truly
missing bit is attestation of the signing key (as far as I can tell).
The rest is there, just badly managed.
> It's not like I and others hadn't given them sufficient prodding
> on it earlier, e.g.:
> http://blog.linuxmint.com/?p=2361#comment-93804
Yeah, it's like Clem either didn't understand your point or ignored it.
> And ... https? Yes, wouldn't have done anything for *this* particular
> issue, but does generally thwart *other* problems.
Out of curiosity, what specific use-case (applies to this particular
example)?
On many Web sites, encrypted transport buys little for lack of need for
confidentiality, and authentication is either relatively unimportant or
is better achieved in other ways for relevant content (e.g., gpg-signed
checksums for ISOs).
Date: Mon, 22 Feb 2016 08:38:01 -0800
>From rick at linuxmafia.com Mon Feb 22 08: 8:01 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: Re: [sf-lug] https - "improves"(?) security?
Organization: If you lived here, you'd be $HOME already.
Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
> Use of https just raises the bar, at least somewhat, on security.
> [...]
> E.g. it makes man-in-the-middle attacks more difficult.
> [...]
All of this is of course true and worth noting.
However, as a reminder, what I asked was: What specific use-case
applies to this particular example (the linuxmint.com Web site
discussed)? I.e., for what content on the site, and what usage
modalities, would confidentiality and/or authentication be aided by
transmission over https, as opposed to http?
The point is, except for Linux Mint isos (authenticated by signed
checksums, albeit that is currently badly done), all content on the site
gets anonymously browsed, everything is non-confidential, and little is
at stake. E.g., the various linuxmint.com sites are not ones for which
man-in-the-middle impersonation -- which requires substantial resources
to either hijack DNS or compromise key routers -- is even remotely
attractive.
This is what I meant when I said 'On many Web sites, encrypted transport
buys little for lack of need for confidentiality, and authentication is
either relatively unimportant or is better achieved in other ways for
relevant content (e.g., gpg-signed checksums for ISOs).' My question
amounted to: What content on linuxmint.com am I missing, for which
https auth & privacy would be even particularly relevant/useful?
Answer turns out to be: none. There is a vague _general_ benefit to
https (as always, and at the cost of some performance and reachability,
e.g. over low-bandwidth connections), but nothing _specific_ on that
site, and no use-case for it, gets any significant benefit.
> Though, in many cases, https adds little to nothing.
Which was my point.
> Also, these days, pretty easy - and *free*! - to do https.
True but not responsive to the question.
> This could also have side benefits of making it less probable that,
> e.g. mirrors, would pick up and display invalid (e.g. "hacked")
> data/information and/or that users would get or download "hacked" or
> tampered with data.
Be serious, now. Anyone who can cache-poison or otherwise hijack users'
DNS or compromise key Internet routers isn't going to waste time doing a
MitM attack against linuxmint.com. Against an online bank, now sure.
Not a Linux Web site that doesn't even handle money.
> On the other hand, if the site puts up hashes of ISOs on their
> website, but doesn't bother to sign the ISOs, I'm not going to
> particularly trust those hashes if they're only offered via http,
> whereas if they're offered via https ... well, a modicum of trust
> above http only.
What makes you think a MitM site cannot have a valid https cert?
Probably from DigiNotar, or the Iranian state CA. ;->
Date: Sun, 21 Feb 2016 17:11:32 -0800
>From rick at linuxmafia.com Sun Feb 21 17: 1:32 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: Re: [sf-lug] Linux Mint iso files hacked.
Organization: If you lived here, you'd be $HOME already.
Quoting Bobbie Sellers (bliss-sf4ever at dslextreme.com):
> Full story and comments(read them) at:
> <http://betanews.com/2016/02/21/linux-mint-hacked-iso-image-compromised/>
As a followup, let me restate my main point in non-technical terms:
The article said 'A bad thing happened with Linux Mint, and you should
avoid this particular bad thing.' _But_ the article told you absolutely
nothing about how to prevent/avoid countless other bad things just like
it. Nor did it attempt to explain what really happened.[1]
Even though _just_ a routine checksum recalculation upon download was
good enough in this case (and almost all cases).
This is an endemic problem with the IT press. They're always about
'Here's news of a thing', but with zero attempt to understand and
explain.
Checksums, yay.
Signatures, yay.
[1] On the bright side, betanews.com did link to
http://blog.linuxmint.com/?p=2994 for more information, so that's good.
Date: Sun, 21 Feb 2016 18:47:12 -0800
>From rick at linuxmafia.com Sun Feb 21 18: 7:12 2016
From: Rick Moen <rick at linuxmafia.com>
To: sf-lug at linuxmafia.com
Subject: Re: [sf-lug] (forw) Re: How to check distro checksums and
signatures
Organization: If you lived here, you'd be $HOME already.
Quoting Bobbie Sellers (bliss-sf4ever at dslextreme.com):
> However in the Linux Mint instance the checksums of the forged
> disks were also hacked so that when you thought you were getting
> the data from the Mint site you were getting it from the forged site.
Not precisely.
Quoting Clem on the blog (http://blog.linuxmint.com/?p=2994): 'What
really helps here is duplication and the community. We were alerted very
fast and we were able to be alerted because people could find
contradicting MD5s (and that\342\200\231s mostly because the MD5s
+aren\342\200\231t just in
one place, but in many).'
This is why I said that checking the mdsums caught the impersonation. I
didn't want to go into _even longer_ detail about how tampered md5sums
occurred in one place but not in all the others, as my post was too long
and detailed as it stood -- and I figured people wanting all the details
could follow links.
> Even magazines can have bad disks and the creator of the distribution
> will say that the checksum is not supplied with the disk because the
> magazine or other publisher may have altered the supplied material.
In my experience, not only are the CDs/DVDs supplied with magazines and
books often meddled with by the publisher, but also they most often
furnish obsolete software.
Honourable exceptions include the annual CeBIT (international computer
expo / trade fair in Hannover) edition of Knoppix, produced for CeBIT by
Klaus Knopper directly and bundled with the issue of _Linux Magazine_
that covers CeBIT.
More information about the sf-lug
mailing list