[sf-lug] How to check distro checksums and signatures (was: Linux Mint iso files hacked.)

[Rick Moen]

Quoting Bobbie Sellers (bliss-sf4ever at dslextreme.com):

> Linux Mint iso files hacked!
> IF you downloaded a Linux Mint iso on February 20, 2016
> It may be compromised with a backdoor.

Bobbie, first, thank you for passing along that news item.

At the same time, that article's contents isn't within a country mile of
the most important lesson to learn, but I can help:


What you should ask yourself is:  What about _any_ downloaded program
(and distribution isos are collections of many programs) makes it worth
trusting?  How do you have reasonable assurance that a program you
retrieve isn't a trojaned fake someone inserted into the download
process?

Linux distros use checksums and cryptographic signatures as a safeguard
-- and you should make sure you check those.

When you use package managers to upgrade/install distro-produced
packages, the process happens transparently to you.  Your running distro
has a crypto 'keyring' that it uses to check signatures on packages, and
the package tools also verify checksums to make sure the downloads are
complete and intact.


By contrast, when you manually download files (such as isos, the burden
is on _you_ to do that checking manually.

Here's the Linux Mint download page:
http://blog.linuxmint.com/?p=2947 

Page starts out with text introducing Linux Mint 17.3 'Rosa' Cinnamon
Edition, and goes on to a 'Direct download' section (alphabetical list
by country of download mirror sites).  _Below_ that long list of
countries is this bit:

   Signatures (to verify your downloaded ISO):

   MD5SUM 64-bit: e71a2aad8b58605e906dbea444dc4983
   MD5SUM 32-bit: 6e7f7e03500747c6c3bfece2c9c8394f
   Signed Sha256sum signatures [link]

This stuff (above) is absolutely _vital_.  You should always look for
it, and _not_ just go straight for the download link.  Your first
thought at the time of looking-for-downloads should be 'Where are the
checksums, and are they signed?'

Checksums currently come in three flavours.  From weakest to strongest
(degree of cryptographic assurance of correctness) types, they are:

  o  md5sum
  o  sha1sum
  o  sha256sum

A checksum, aka a hash, is a number derived from a file, and much
smaller than that file, that you can use to verify data integrity by 
recalculating the checksum of the file you received and making sure it's
the same as what the download site claims it should be.  If it's a
match, then you have pretty strong proof your file download wasn't
truncated or corrupted in transit.

The utilities to do this task in Linux are named (yay!) md5sum, sha1sum,
and sha256sum.  Check your distro; it'll have them already.

Let's say you download linuxmint-17.3-cinnamon-64bit.iso.  OK, now you
immediately do:

$ md5sum linuxmint-17.3-cinnamon-64bit.iso
If your download was complete & uncorrupted, /usr/bin/md5sum will respond:
e71a2aad8b58605e906dbea444dc4983
(The checksum tools also have a '-c' option to do the comparison for you
more automatically.)


Verifying the checksum is step #1 of 2.  The second step is checking the
gpg[1] signature of the checksum -- because, unfortunately, while
recalculating the checksum verifies completeness, it doesn't verify
authenticity, because the same Web site intruder who modifies Linux
Mint's page to post links to fake isos might also alter the page to
specify fake checksums.

So, you use /usr/bin/gpg to verify signature of the posted checksum.  I
was going to show you how to do that with Linux Mint, but
unfortunately it seems they muffed this information:  As you'll see
in the quoted 'Signatures' text (above), they posted md5sums but not
_signatures_ of those md5sums.  They published signatures of sha256sums,
but not the sha256sums themselves.

This post is already long enough, so I won't invent an example of gpg
checking.  Perhaps some other time, or someone else might speak up to
show how one does it.  Example tutorial:
https://help.ubuntu.com/community/VerifyIsoHowto


You should be asking yourself, 'OK, if checksums check competeness but
not authenticity, and signing is used to check authenticitity, what 
checks that the signatures are authentic?  Can't the same intruder who 
modifies a distro download page to post links to fake isos and fake
checksums also modify it to show fake gpg signatures?'

Yes, indeed, and this is where signatures being known or not is useful.
If the key used to sign a distro's isos suddenly changes, many people
who've known and relied on that key will quickly notice and raise the
alarm.  Also, previously unseen gpg keys can be checked by using gpg to
note who has vouched for them (signing those keys with their own keys).

In this case, the alarm got raised quickly because Linux Mint's md5sums 
aren't published in just one place but in many, and downloaders checking
the fake isos (in many cases) got unmatching checksums.

How did the compromise occur?  It occurred because the affected server
runs WordPress, which IMO has terrible security, just like many other
popular PHP Web applications.  (PHP, IMO, is a security menace
generally.)  Clem Lefebvre of Linux Mint says:  'We found an uploaded
php backdoor in the theme directory of a wordpress installation, which
was 1 day old and had no plugins running. The theme was new but most
importantly I think we had lax file permissions on this.'

The intruders were then able to run OS shell as user 'www-data', the
user that runs Apache httpd.  Note:  If you run a Web server, it is very
important as a fallback security measure to ensure that user www-data
has restricted privileges.  Even then, hostile processes running as that
user could do serious mischief, and are a serious matter.


I just posted this to Clem's blog page http://blog.linuxmint.com/?p=2994
(awaiting moderation):

  Clem, your point is good one that duplication and the community was an
  effective cross-check and instrumental in spotting the compromise
  quickly. But (and you knew there was a ‘but’, right?) the people on this
  thread suggesting improved gpg-signing of checksums also have a valid
  point.

  You said ‘You can find them at
  http://ftp.heanet.ie/pub/linuxmint.com/stable/17.3/ also along with
  signed sha256sums’ and ‘we’ll probably default to showing sha256 for
  upcoming releases’ — which is good news. However, please note that
  primary download pages such as http://blog.linuxmint.com/?p=2947 have
  for a long time (and still) listed md5sums and gpg signatures of
  sha256sums, but not included gpg signatures of md5sums, and not included
  sha256sums. So, unless a member of the public thinks to also look on
  http://ftp.heanet.ie, he/she could not easily check gpg signatures at
  all.

  I would like to politely suggest that you good folks take a careful look
  at the published means of verifying authenticity, and make sure
  everything works even for half-clued outsiders, and that this include
  care to make sure signing keys are publicised and able to be vetted
  using the gpg chain of trust.

  Thank you for Linux Mint, and for your good work.

  Best Regards,
  Rick Moen
  rick at linuxmafia.com


Well, OK, since this post is already windy, I'll chance a bit more breeze:  
Let's say you somehow stumbled across 
http://ftp.heanet.ie/pub/linuxmint.com/stable/17.3/ that has _both_ the 
sha2sums and the gpg signature for those sums.  You fetch
sha256sum.txt.gpg and sha256sum.txt to your local disk.  Then:

  $ gpg --verify sha256sum.txt.gpg sha256sum.txt
  gpg: Signature made Wed 06 Jan 2016 08:06:20 AM PST using DSA key ID 0FF405B2
  gpg: Can't check signature: public key not found
  $

This is not surprising, because your system (or at least mine) hasn't 
previously encountered the gpg with 'DSA key ID 0FF405B2'.  (That is 
a hash of the Clem Lefebre's signing key by the way, a hash.)
So, you ask the public keyservers about that key:

  $ gpg --search-keys 0FF405B2
  gpg: searching for "0FF405B2" from hkp server pool.sks-keyservers.net
  (1)     Clement Lefebvre (Linux Mint Package Repository v1) <root at linuxmint.co
          1024 bit DSA key E1A38B8F144675D060EA666F3EE67F3D0FF405B2, created: 2009-04-29
  Keys 1-1 of 1 for "0FF405B2".  Enter number(s), N)ext, or Q)uit > 1
  gpg: requesting key 0FF405B2 from hkp server pool.sks-keyservers.net
  gpg: key 0FF405B2: public key "Clement Lefebvre (Linux Mint Package Repository v1) <root at linuxmint.com>" imported
  gpg: no ultimately trusted keys found
  gpg: Total number processed: 1
  gpg:               imported: 1
  $

And now, to no great surprise, the signature checks against the public
key you just fetched.

 $ gpg --verify sha256sum.txt.gpg sha256sum.txt
  gpg: Signature made Wed 06 Jan 2016 08:06:20 AM PST using DSA key ID 0FF405B2
  gpg: Good signature from "Clement Lefebvre (Linux Mint Package Repository v1) <root at linuxmint.com>"
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: E1A3 8B8F 1446 75D0 60EA  666F 3EE6 7F3D 0FF4 05B2
  $

In the 'WARNING' text, gpg is properly paranoid, pointing out that you
have no special reason to think the public keyserver's entry is actually
Clem's key:  gpg couldn't find any signature _of_ Clem's key to furnish
an additional reason to trust it.  However, you can reasonably expect
that if the public keyservers had a fake key for Linux Mint and Clem
Lefebre, people would notice quickly.


If checking signatures seems too complex, _at least_ verify (recalculate
after download) the checksums -- which is easy.  In the current Linux
Mint case, for example, verifying checksums was sufficient to catch the
forgeries.


> Full story and comments(read them) at:
> <http://betanews.com/2016/02/21/linux-mint-hacked-iso-image-compromised/>

You know, Bobbie, intending no criticism of you personally, but the
above news story provides nothing whatsoever about how to detect 
forgeries.  It just says 'beware of this one forgery'.

Above I've attempted to fix that grievous omission.

Please, folks, don't just download executables (including isos) and
trust them.  Stop that, please.  Use checksums.  Use signatures.

> I downloaded the iso images of Mint that I have at the meetings long
> before this happened, if anyone has any concern.

Yes but also:  Use checksums.  Use signatures.


[1] 'gpg' is the open source reimplementation of PGP, Phil Zimmermann's 
famous Pretty Good Privacy program.  Be aware that gpg has an
infamously confusing and opaque command-line interface.  Don't hesitate
to Web-search for tutorials such as the Ubuntu one I cited above.