From rick Tue Sep 24 22:19:26 2002
Date: Tue, 24 Sep 2002 22:19:26 -0700
To: kmself@ix.netcom.com
Subject: debsigs
User-Agent: Mutt/1.4i

So, I made two dumbass errors: debsigs is the packager's tool to sign packages. The one you install to autocheck package signatures is debsig-verify. Also, IMPORTANT: You also must comment out the "no-debsig" line currently in /etc/dpkg/dpkg.cfg by default. (Turning it on may cause legitimate packages to get rejected, if they weren't signed correctly or had other irregularities. Because few people yet try to use this option, I suspect you might encounter many such glitches, but haven't verified.)

Apologies for getting that wrong.

Points addressed below:

1. Security threat via Debian package mirrors gets blown way out of proportion because people don't understand the threat models. Which accordingly I don't lose sleep over. Herewith discussed.

2. Problems inherent in package signing are generally underestimated, for the same reason. Likewise discussed.


Biggest potential security exposure mode via the mirror system is root compromise of the ftp.debian.org host. This would then likely compromise downstream mirrors as they perform rsync mirroring from it. ftp.debian.org is the "root" of the mirror tree. Remedy is that scrutiny of this host is maximal. Working assumption is that compromise is unlikely; detection would be quick. Effects would nonetheless be a bit messy.

Second threat mode is compromise of a downstream mirror. Divides into cases of a) intruder disables daily rsync mirror session, or b) doesn't. In case a, exposure to machines polling for packages may persist as long as admin fails to notice his mirror... isn't. In case b, unauth packages get auto-erased every mirror run. I suppose ongoing resupply of same would be theoretically possible.

Third threat mode involves intruder becoming an official package maintainer (requires, among other things, presenting positive ID at a Debian keysigning and being known personally by existing Debian maintainers who sign his key), and then submitting system-subverting .debs signed by himself. Which signatures go into the .changes file on ftp.debian.org, and retained but not sent out to the public. Similarly, uploaded source packages generate stored signatures from the .dsc files, and also stored in the archive. (ftp.debian.org will not accept any .deb or source file without a gpg signature that's on record in the Debian maintainers' keyring.) An intruder might thus be able to send up nasty-surprise files once, but the Debian mafia would, so to speak, know where he lives: There's no such thing as an obscure, hidden, anonymous, or not well-identified Debian package-maintainer. And peer review should catch the nastiness pretty quickly.

There's some reason why merely subverting DNS and posing one's EvilCo host as a supposed official Debian mirror -- or some similar man-in-the-middle attack -- isn't practical, but I can't remember details.

The above picture is why I don't sweat things like debsig-verify, etc. It's just not much of a threat. (Other candidate threat models have been discussed to death and found implausible.)


Existing signing architecture:

All Packages[.gz] files contain md5sums for every .deb listed. The former files of course get refreshed whenever you do "apt-get update". apt-get [apt-zip, apt-cdrom] refuses to further handle any package whose md5sum proves upon download to not match the stored hash value. Instead, it will attempt redownload, on the assumption that it wasn't fully retrieved.

Each Packages[.gz] file's own md5sum is provided in the accompanying Release file.

Release files are in turn signed by the master package-releasing program's gpg key, and the hash stored in Release.gpg in the same directory.

Automated methods for checking Release.gpg files aren't yet in place. A script exists, but hasn't yet been disseminated because there are still some fundamental holes to deal with, e.g., EvilCo gets root on a mirror, and merely fails to roll in security updates. debsigs will still check out, and even Release.gpg will still be valid -- just outdated. But systems using the mirror will fail to get security updates that EvilCo makes sure the mirror isn't providing.

Package debian-keyring is (pardon the pun) key to all this, as it's the canonical list of authorised package signers. You have to get updates more often if you're on unstable/testing than if you're on stable.

[02/2004 update: apt version 0.6 adds the capability of checking Release.gpg files against debian-keyring, and enables it by default.]

debsums: Some .debs (not many, yet?) ship with md5sum values of all included files. "debsums -ca" is the proximate equivalent of "rpm -Va" (listing all changed files), except that the rpm equivalent has to exclude conffiles.

'Course, if you're going to absolutely rely on those sorts of signing techniques to protect you against hostile package mirror sites (regardless of your distribution), you'd better make sure you check all packages using only package-handling and signature-verification tools stored on read-only media, booted from a trustable system in single-user mode. And nobody does that.

Partly because the system's still being thought out and inherent holes such as the those cited being pondered, the system isn't yet fully implemented. Discussed in this thread:

http://lists.debian.org/debian-securitty/2001/debian-security-200112/msg00239.html
(Discussion is a little outdated.)

Anthony Towns has packaged several small utilities by Fruhwirth Clemens with a wrapper program of his own (apt-check-security-sigs) inside a "secpack" package that automatically verifies signatures before installing package upgrades: http://therapy.endorphin.org/secpack/


Date: Wed, 25 Sep 2002 12:36:17 -0400
From: Joey Hess (joeyh@debian.org)
To: debian-security@lists.debian.org
Subject: Re: How reliable is "debsums"?
User-Agent: Mutt/1.4i

Justin Ryan wrote:
> Use both! One advantage of debsums is that you can compare md5sums
> against a package, rather than just the system db.

If you already have a .deb file to compare, you don't need debsums, just use this command:

joey@dragon:~/bin$ cat verifydeb
#!/bin/sh
dpkg --fsys-tarfile $1 | tar -C / -d

--
see shy jo


From rick Fri Sep 27 15:20:41 2002
Date: Fri, 27 Sep 2002 15:20:41 -0700
To: plug@lists.q-linux.com
Subject: Re: RPM and intrusion detection WAS>> Re: [plug] A short and sweet Slackware package tutorial
User-Agent: Mutt/1.4i

Quoting Ian C. Sison (ian.s@qsr.com.ph):

> Not wanting to go into another set of detail picking on this issue, i'd
> simply want to say that MD5 checking effectivity is accurate on a case to
> case basis. It will give you on occasion an idea that a box has been
> rooted, and it will show you just which binaries and config files may be
> compromised, but as with any other solution, is not the end all and
> be-all of intrusion detection.

Agreed -- if and only if you have reason to trust (1) the checksum-verification tool and the environment it runs in, and (2) the checksum database.

In other words, if you check your package database (without looking after those troublesome details) and see clear signs of funny business going on, you can probably rely on that and call the warning A Good Thing. But a clean result doesn't mean jack.

Let me say that again: A clean result doesn't mean jack.

I got barred from the mailing list of a California user group (NBLUG) dominated by semi-clued Red Hat users, immediately after making a sarcastic remark about their self-delusion on this matter[1], but it's an important truth: People who just do "rpm -Va" and conclude they're not root compromised are kidding themselves. Others who advise them to do that are not their friends.

And, of course, my point is that I see both of those things all the time. I was, in effect, thrown off that mailing list for exposing the fact that the group was dispensing horrifically bad security advice to the unwary. The clique in charge enjoyed being looked up to, by raw newcomers.

> Even a total reinstall from trusted media can give you a false sense of
> security. If what you did is just simply reinstall, and copy over (for
> instance) of your web pages with cgi scripts which contain
> vulnerabilities, then you will be rooted again.

I've said it before, and I'll probably have to say it again: You cannot trust either the executables or configuration files of a compromised system.

>> There are lots of reasons. For one thing, the intruder's actions might
>> involve only your configuration files, not your binaries. Your MD5 sums
>> don't cover the former, do they? If they do, that would be very
>
> Yes they do.
>
> An 'rpm -Va' will give md5 sum failures for files such as inittab or
> httpd.conf if you even modified it from its original state.

This is not the whole truth: There is an RPM build option to omit configuration files from the stored md5 values. Some RPM packages are built with that option, others are not. Those who use it are doing so largely because of people getting alarmed at normal, local-system changes to configuration files.

What this means, in effect, is that comparing conffiles' md5sums against those of initial default versions that were immediately discarded following installation isn't particularly useful. (Compare AIDE, Tripwire, Integrit, and the like -- which more usefully fill that role.)

>> And that raises the question of where you have the MD5 sums stored.
>> Would that be on the host itself, on media that aren't hardware
>> write-protected? Like, for example, /var/lib/rpm/* ? If so, what
>> reason do you have to trust that those MD5 sums haven't been changed by
>> the intruder?
>
> The only way an intruder can change that is to use RPM itself to 'update'
> packages on your system.

That is absolutely incorrect, and I'm disapointed that you would claim that. The RPM database is a simple BerkeleyDB database! This is documented. It's not some super-secret protected storage facility.

>> And what do you run to check those MD5 sums? A utility that's on the
>> compromised system? What earthly reason do you have to expect it to
>> give you results that haven't been specified by the intruder? Pretty
>> much the only way you can avoid that risk is to boot from
>> non-compromised media (e.g., a CDR or maintenance floppy) and run
>> programs entirely from it.
>
> Which is what something like chkrootkit suggests. Again, it's another way
> of detecting intrusion, and it's not a total solution. It's just one of
> the tools you use.

So, to reiterate: A clean "rpm -Va" result doesn't mean jack. That's imaginary, toy security.


[1] Replying to one of the LUG leaders, Red Hat advocate Eric Eisenhart:

> Problem with this is, it's possible that somebody might have
> installed a rootkit that also changed the RPM database or the RPM
> program or the kernel to see things as being as they still should
> be.

<deadpan>
That's why all Red Hat users store safety copies of /var/lib/rpm/* off-system, right?
</deadpan>

This seems to be what got me kicked off and locked out.*

-- 
Cheers,      "On the face of it, Microsoft complaining about the source license 
Rick Moen    used by Linux is like the event horizon calling the kettle black."
rick@linuxmafia.com             -- Adam Barr, former Microsoft Corp. programmer


* 2004 note: Eisenhart often claims (almost always behind my back) that it was for "flaming". All 56 of my posts to that mailing list can be seen at http://linuxmafia.com/~rick/linux-info2/nblug-talk.html . I have an outstanding $100 offer to Eisenhart if he can convince a neutral third party that those posts thus archived contain "flaming".

Why this happened, especially with zero warning, still seems a little mysterious -- but the group was at the time (but no longer is) militantly Red Hat-centric (fittingly, it was the only Northern California stop on that company's 2002 Road Tour), and seemed to powerfully resent knowledgeable Linux users who didn't share their view.


From rick Sun Sep 29 15:27:21 2002
Date: Sun, 29 Sep 2002 15:27:21 -0700
To: plug@lists.q-linux.com
Subject: Re: RPM and intrusion detection WAS>> Re: [plug] A short and sweet Slackware package tutorial
User-Agent: Mutt/1.4i

Quoting Ian C. Sison (ian.s@qsr.com.ph):

[RPM database:]

> ... which the most common of rootkits don't care to touch anyway.

Which is a good point. For now. But please note that the most significant fact about scripted, automated attack tools is that they lower drastically the amount of intelligence and technical competence required to exploit vulnerabilities (which is why we refer to "script kiddies"). This was perhaps the most important occurrence in the security field, in the 1990s: One extremely clever person, who does have a deep understanding of vulnerable systems, develops and packages an easy-to-use kit for breaking into particular sorts of systems and hiding one's presence. He sends that kit out, where it's picked up by hundreds of thousands of bored kids to use worldwide, who don't need to even understand what they're doing.

Thus far, to my knowledge, nobody has bothered to script intruders' modifications to RPM databases, probably because such a vast number of vulnerable systems are run by completely unwary admins that the extra stealth would be not worth the coding effort. But all it would take is for one clever person to go to that extra bother.

Certainly, doing rpm -Va is a worthwhile step in case it finds positive evidence of break-in.

> Simply reformatting or reinstalling the system immediately does not
> help, because you should find out the weakness in your security or it
> will happen again.

Agreed.

-- 
Cheers,
Rick Moen                                        This space for rant.
rick@linuxmafia.com

Date: Fri, 18 Oct 2002 16:54:31 +0200
From: Jan Niehusmann (jan@debian.org)
To: debian-security@lists.debian.org
Subject: Re: Automatic Debian security updates, an Implementation
User-Agent: Mutt/1.4i

On Fri, Oct 18, 2002 at 10:48:16AM -0400, R. Bradley Tilley wrote:
> Why can't apt-get be modified to check the md5sum of a package against an
> official debian md5sum list before downloading and installing debs? This
> seems much simpler and easier than signing debs.

It does. The problem is, how to get an official Debian md5sum list? This is, basically, what apt-check-sigs does. It checks the validity of the Packages files (which contains md5sums of individual packages) with a gpg signature.

Jan


From: Gustavo Franco (stratus@legolas.alternex.com.br)
To: debian-security@lists.debian.org
Subject: Re: Automatic Debian security updates, an Implementation
X-Mailer: Ximian Evolution 1.0.5
Date: 18 Oct 2002 09:55:03 -0300

On Fri, 2002-10-18 at 09:33, Mark Janssen wrote:
> On Fri, 2002-10-18 at 14:24, R. Bradley Tilley wrote:
>
>> Can someone explain why 'apt-get update && apt-get dist-upgrade' is not
>> sufficient to keep a debian system secure and updated?
>
> It'll get to you when you have 200+ Debian systems spread across the
> Internet in different cities, timezones and administrative domains :)

You can try cron-apt package[1] and apt-check-sigs[2] to do it! Now, I've twelve servers running Debian GNU/Linux and I'm using one apt-proxy[3] and aptwatcher(like cron-apt).

[1] = http://packages.debian.org/cron-apt/
[2] = http://people.debian.org/~ajt/
[3] = http://apt-proxy.sourceforge.net/

Talking about secpack, is it non-free? I can't see in your mail (Clemens) the url or apt-line to get the source package.

Thanks,
--
Gustavo Franco -- stratus@acm.org


Date: Fri, 18 Oct 2002 15:55:53 +0200
From: Jan Niehusmann (jan@debian.org)
To: debian-security@lists.debian.org
Subject: Re: Automatic Debian security updates, an Implementation
User-Agent: Mutt/1.4i

On Fri, Oct 18, 2002 at 08:20:14AM -0500, Joseph Pingenot wrote:

> If people are interested enough in it, I might throw together something
> more formal.

IMHO there is no lack of interesting ideas - what we really need are implementations.

apt-check-sigs is a nice proof-of-concept, and the debsigs stuff could also improve security significantly. Together, I'd say they'd suffice to make the debian mirrors extremely tamper-proof.

But apt-check-sigs is lacking nice integration into existing tools, and debsigs doesn't really work, because packages are not signed, which is IMHO caused by inappropriate helper tools at packaging time.

So implementing these tools, and then changing policy to make package signatures mandatory, seems to be the most feasible approach.

Writing new proposals for advanced security schemes doesn't help and may even delay implementation of working mechanismns.

Jan


From: andrew lattis (alattis@mindspring.com)
To: debian-security@lists.debian.org
Date: Tue, 22 Oct 2002 11:13:41 -0400
Subject: Re: AIDE Information Overload
User-Agent: Mutt/1.4i

On Tue, 22 Oct 2002, Arthur de Jong wrote:

> Apart from that I also use tools like debsums to keep me informed of
> integrity (although a lot of packages don't provide all or correct
> md5sums). (Maybe I should file some bugreports for wrong md5sums.)

You also might want to check out tiger. It will run debsums, and can check your currently installed packages for security advisories, as well as a few other general security checks -- although I've never seen an e-mail about security advisories, so I'm either missing something or am really quick to update....

I like the idea of bugreports for missing md5sums: A few I'd really like to see are sysvinit, bash, dpkg(?)


From: Fruhwirth Clemens (clemens@endorphin.org)
To: debian-security@lists.debian.org
Cc: stratus@legolas.alternex.com.br
Subject: Re: Automatic Debian security updates, an Implementation
User-Agent: Mutt/1.3.28i
Date: Tue, 19 Nov 2002 21:45:02 +0100

On Fri, 2002-10-18 at 09:55, Gustavo Franco wrote:

> Talking about secpack, is it non-free? I can't see in your mail (Clemens)
> the url or apt-line to get the source package.

No, it's BSD. I didn't dare to put up a license for that minimal collection. There isn't even a source package. I just dpkg-deb -b the few files with a DEBIAN/control file. See http://therapy.endorphin.org/secpack/

Regards, Clemens

P.S.: Sorry for replying that late, but someone obviously removed my Cc line and I haven't been subscribed to debian-security. Just found your message accidently in the archives.


[Excerpt from a Debianplanet discussion:]


Subject: onboard signatures insufficient
Author: robot101
Date: Wednesday, 2003/11/26 - 10:10

This has been rehashed lots of times on various Debian lists, and I imagine JoeBuck is aware of this, but including signatures with each package is not sufficient to ensure you are not being decieved by your mirror. Why? Because outdated, insecure and exploitable packages which have been superseded by newer versions in stable/unstable/security/whatever still have valid signatures. This is why no built-in signature system has been made for debs, because the Release signature is more correct and we already have it. It states that at the time the signature was made, these specific packages constitute the latest version of whichever release. An onboard signature just says "at some point a debian developer made this package", and that doesn't cut it.


Subject: buildd signatures not automated
Author: robot101
Date: Thursday, 2003/11/27 - 19:59

Buildd maintainers manually intervene to batch-sign the .changes files generated by their buildd systems. I don't think it's fair to infer from this that the buildd maintainer is responsible for the content of all these packages, just that he is alleging that the binary packages contain a true representation of what was in the source package. This is also an argument against embedding the .changes (ie build time) signatures into the .debs. It's far far more meaningful and important for Debian to get the automated checking of Release signatures sorted out. Anything else (debsigs, whatever) is just a mere sideshow to this.


Subject: This is already done too
Author: robot101
Date: Thursday, 2003/11/27 - 19:52

When you generate a package, you also get a .changes file which is signed. If you're distributing packages outside of Debian, it's common to make a signed-but-invalid-for-upload (target an imaginary distribution for example) .changes file available alongside your packages and sources. Or you could sign the Packages file yourself or generate a Release file and sign that, or whatever took your fancy.

Once packages are uploaded to Debian, .changes files are checked for valid signatures and retained (although not publically available) indefinitely (to the best of my knowledge). We can always find out who uploaded what.


Subject: To a point...
Author: robot101
Date: Wednesday, 2003/11/26 - 10:06

Apt won't install a package if its md5sum is different to the one advertised in the Packages list. That's a good thing, it means that if we can check signatures down to Packages, apt will do the rest. apt-secure or apt-checksigs (IIRC) can do the steps from Release.gpg down to this.

Subject: hmmmm
Author: robot101
Date: Wednesday, 2003/11/26 - 10:17

While (as demonstrated with trivial application of the pidgeon hole principle) it's possible to demonstrate files that have the same MD5sum, I would be exceedingly interested to see a file that had the same MD5sum as a deb and was also a valid Debian package. Let alone apparently the same package with an added trojan. Yes, it's possible, but intensely unlikely.

As I state elsewhere on this page, an onboard signature of a package isn't sufficient to ensure the security and currentness of a set of packages that form a release. This is why Debian doesn't and will probably never have built in signatures, and also why we already have Release.gpg signatures.

All apt-secure is about is letting you define a few keys you trust, and requiring valid Release signatures from these keys for each entry in sources.list. Key wise, there is one per Debian release, stable keys being signed by the release manager, and testing/unstable keys unsigned because they are used unattended and are only as secure as our ftp master server is.

Regards,
Rob