Constructive Paranoia at the End of 2003

By Rick Moen

Some weeks ago, I was spending some time assisting my mother-in-law, who's working on her Ph.D in computer network security, do a survey of a half-dozen or so Linux users at local user group, CABAL, about our security practices — with results like these:

Do we use anti-viral software? (No, except where we handle files or mail destined for Microsoft-OS machines.) Do we study our networks' security exposure using vulnerability-scanning software such as nmap and snort? (Many of us do, yes.) Do we run log-analysis security utilities such as logcheck? (Ditto.) Do we run Intrusion Detection System (IDS) suites such as Tripwire? (Almost never.) What measures do we take to eliminate security holes as they arise in a timely fashion? (Various.) Do we use kernel-level IP-traffic filtering scripts ("firewalls")? (Some do. Several more-experienced users operating all-Unix networks do not.)

These questions kept haunting me as I answered questions from new Linux administrators. Sometimes, those were very astute: Q: "How can you be sure that your system hasn't been compromised by hostile parties?" A: "Excellent question. You can't know, absolutely. A truly subtle and competent intruder manifests those abilities in part by being difficult to spot, and covering his or her tracks. But intruders (or their automated attack tools) generally break into a system to do some significant unauthorised activity, leaving clues that will be spotted by alert and capable admins who know their systems well enough to notice peculiar goings-on." That answer tends to leave questioners slightly uneasy — as intended.

Two Classes of Attack

I always told new admins that there two levels of threat to systems: from outside and from inside. Conventional thinking worries mostly about the former, e.g., the sort of perimeter survey you get by running

nmap -vv -sT -sR -O -n -oN tcpscan.log 10.0.1.3
nmap -vv -sU -sR -O -n -oN udpscan.log 10.0.1.3
nmap -vv -sA -sR -O -n -oN ackscan.log 10.0.1.3

...from the far side of your LAN (a different machine) against your IP=10.0.1.3 Linux host, to guesstimate how hard the latter's (figurative) exterior shell is — in the event of attack from elsewhere.

But the latter sort of threat — from the inside — is both more worrisome and more interesting. That is, if there's a entrance method (ssh, whatever) from the outside into your machine, and someone steals the password or other token required to use it, then you have unwelcome guests, who can then subvert your system's security from its own command prompt. It's well known that the latter step's often successful, but the real news is how easily and frequently passwords get stolen.

Consider inbound ssh access to your machine(s), a convenience much used and cherished by Unix users. Do you ever ssh in from machine you don't personally administer and have absolute confidence in? Even if you don't, do you always carry your ~/.ssh/known_hosts2 file with you, so you can be sure the remote host you reach is really yours? If you're ever lax on any of those matters, and you're even slightly unlucky, the bad guys will steal your access tokens and enter masquerading as you, later.

Even if you never do any of those things (making you a rare paranoic, indeed), can you say the same of all the friends you gave shell accounts to? Nobody ever used a cybercafe or university computer, or used PuTTY on a family Windows box teeming with spyware (or maybe even a keystroke-recording dongle connected to the keyboard)? Thought not. And there's your problem.

The November Surprise

Which brings us to November's security incidents. A timeline should help us set the scene:

You'll notice how quickly the Debian and Gentoo people realised their problem, and corrected it — a point I'll come back to. But the first point to note is how the bad guy entered to begin with — a necessary first step before he/she could use the kernel flaw.

It turns out that one of the 1000+ Debian developers had been the victim of security-token theft. He or she used an ssh client on some machine, somewhere, that happened to have already been subverted. The ssh software was "trojaned" and privately logged his/her Debian-server login credentials, later conveying those to the attacker — who was then able to waltz in as if he/she were the developer. Only then did he or she use the kernel bug to escalate privilege to root-user access, something he/she might equally have done by finding an un-patched flaw in any other piece of security-sensitive software. The main point is: The intruder got in despite everyone (probably) being reasonably cautious and prudent, because one of his or her password-grabbing processes got lucky somewhere.

IDSes and Their Discontents

Two things put an immediate halt to this malarkey at the debian.org and gentoo.org sites. One was the presence of alert sysadmins, who, in debian.org's case, noticed the pattern of kernel "oopses" on two machines simultaneously, judged that far too great a coincidence, and thus were tipped off. (Similar alarm bells probably went through the gentoo.org admins' heads, but far fewer incident details have emerged from them.)

The other was a much-lauded but little-used type of software called host-based Intrusion Detection Systems, the classic example of which is Tripwire, invented by Gene Spafford and Eugene Kim at the fabled COAST security laboratory at Purdue University from 1992 through 1994. For most of its history, it was proprietary (with source code available for inspection but no right to independently develop it), offered for sale to business, and with a "free for non-commercial use" edition called Tripwire Academic Source Release (ASR) available for download.

Over time, Tripwire underwent a complete rewrite that unfortunately did nothing about the program's nagging usability issues (about which, more below), and then, under pressure from open-source alternatives and with help from VA Linux Systems, its sponsoring firm (Tripwire, Inc.) re-released Tripwire in October 2000 as open source software under the GNU GPL.

CTO Eugene Kim has, since then, professed indignation at the sparse participation in response by open source community coders — but the firm's pride and joy turns out to be non-portable C++ with no autoconf support. (Gosh, Gene, maybe those antediluvian coding standards, bizarre choice of language, and your firm's turning to open source only after mindshare had already fled to more-open alternatives have something to do with it?)

I remember Tripwire ASR; like most sysadmins, not at all fondly — having attempted to start living with it in 1994 and deciding it wasn't worth the hassle. It was and is an absolute horror to set up. In theory, you write a description of what files and directories (and what aspects of them) to check for unauthorised changes, have it take a snapshot of the current, non-compromised system state, and commit to disk all that information in a cryptographically verifiable state. Nice theory; teeth-grating execution.

Unfortunately, the tools and configuration syntax are impenetrably obscure, every operation runs incredibly slowly, and its system-integrity-checking mode churns out long and mostly meaningless reports to the root user, which must be studied and then used to further refine Tripwire's human-hostile ruleset to gradually refocus its attention on system changes that actually matter and cease reporting trivia. Because of the heavy use of encryption, each of those steps tends to be dog-slow, and the process must be run through iteratively, many times, using expert knowledge of one's system, before the results start to be useful and not just verbose babble.

Information overload, horrific configuration language, slow and performance-sapping operation, twisted administrative interface... argh! Save me from this! I quit wrestling with it, within a week or two.

Not at all coincidentally, starting one year before Tripwire went open source, it started getting serious open-source competition, starting with AIDE, a package by Rami Lehti and Pablo Virolainen in Finland. AIDE has lately been joined by similar designs starting in 2001: Ed L. Kashin's Integrit, Rainer Wichmann's Samhain, Yoann Vandoorselaere's Prelude IDS, and no doubt others.

I considered using AIDE, when it emerged in August 1999, and played with it a bit. Where Tripwire was slow and system-clogging, AIDE was fast and light. Where Tripwire was obscure and prone to breakage with puzzling errors, AIDE was easy to understand and debug. It had one big problem: The system-snapshot database, program binary, and configuration file weren't stored with cryptographic verification (as Tripwire does). The authors urged, instead, that those all be stored on write-protected media and updated only as needed.

Keeping the AIDE files on floppy or CDR is a major nuisance. The alternative, of just using them on the system's own hard drive, is easier but tends to give a false sense of security. That is, if/when the bad guy comes in and subverts your system, isn't he or she going to subvert the IDS, too? So, when an IDS tells you all is well, how do you know the bad guy isn't pulling its puppet strings? Tripwire has an answer to that objection; AIDE and friends do not.

That sort of false reassurance is the same one often encountered among users of RPM-based systems reassured by the results of running "rpm -Va" to "verify" the md5sum signatures of installed files: The values are "verified" against a simple Berkeley DB record in /var/lib/rpm — which of course a competent intruder will update to match his or her changes.

So, in the end, I didn't run AIDE routinely. The Debian Project developer boxes did, and it paid off — the intruder having been sophisticated enough to leverage a previously-unknown Linux kernel exploit, but not enough to notice AIDE and sandbag it before it could inform on him/her.

The Best of Both Worlds

Following November's security incidents, and my mother-in-law's raised eyebrow over us Linux old-fogies not running host-based IDSes, I felt I had to revisit the matter, and explore options. To my great fortune, the last piece serendipitously arrived in a post to the debian-security mailing list by Lupe Christoph:

"We don't use AIDE exclusively at a client site, but in combination with Tripwire. We think Tripwire is a little more secure because it uses signed databases. So, we protect aide.db with Tripwire. AIDE is used for the parts Tripwire can't do because of its limited configurability...."

Um... yeah. Why didn't I think of that? Whacking Tripwire's configuration down to just the few minimum items it's best suited to handle, including AIDE's own otherwise-unchecked files, means the usual pain of using Tripwire fades into background noise, and makes its operations run in less than geologic time. Meanwhile, AIDE picks up the rest — and I don't have to worry that I'm fooling myself into complacency like an overconfident rpm user. It works a treat.

Concluding Sermon

My Web site's "Lexicon" page includes Moen's Laws, such as:

Moen's First Law of Security: "It's easier to break in from the inside." E.g., many Internet break-ins result from masquerading as a legitimate user to gain user-level access, e.g., with sniffed passwords. The attacker then has a dramatically wider field of system weak points he or she can attack, compared to penetrating the system from outside.
Moen's Second Law of Security: "A system can be only as secure as the dumbest action it permits its dumbest user to perform." Your users are often your weakest link; smart bad guys will attack you there (e.g., via social engineering). Smart admins will try to compensate for this tendency, e.g., by using multi-factor authentication instead of just passwords.

Between the two of those, one could have predicted the sort of small calamity that overcame the Debian, Gentoo, and Savannah projects in November. Given the considerable likelihood of security tokens being stolen, especially on machines used by many people, it's a wonder it didn't happen sooner. The small miracle of that was that two of the three detected and fixed the break-in immediately — courtesy of host-based IDSes.

Detection is great, and better than a kick in the head (or living in a fool's paradise), but what about prevention? One way is to run an ssh daemon on an additional, non-standard port (maybe 2222 instead of 22) that requires OPIE or S/Key one-time passwords instead of regular, stealable ssh authentication. More precisely, one-time passwords can certainly be stolen, but then are useless because they've already been used up by the authorised user.

One-time passwords are a nuisance to manage: You generate a password "seed" and convey them somehow to your user. He or she either carries around a printout in very small type of the resulting series of 500 or so one-time passwords, crossing them off as they're used up, or puts the seed in a PalmPilot and generates those passwords from it using PalmKey, Strip, or pilOTP for PalmOS.

I may not use such a setup every time I'm away from home and tempted to cut corners — nor require my users to — but it might be nice to have that option the next time I'm in a cybercafe or some malware-infested bank of public Windows machines at a trade show.

Some Protective Measures to Ponder:

Wichert Akkerman's page of information on the Debian.org compromise includes some intriguing recommendations to add to that, including some behavioural ones:

The first of these is interesting and subtle: How many times have you ssh'd to someone else's machine, and then scp'd a file back to yourself ("pushing" it back to yourself)? Well, don't do that. Instead, scp it in "pull" mode from your own machine's command line:

$ scp username@remotehost:/tmp/somefile .

...rather than this form on remotehost's command line:

$ scp /tmp/somefile username@myhost:

Why? This gets back to the problem of stolen tokens, again: When you initiate the scp from remotehost to "push" the file back to where you are on myhost, you have to provide a stealable security token on a machine you don't control and have no reason to trust. "Pulling" the file from myhost poses no such risk.

Hardly anyone follows Wichert's second recommendation (unique passwords) because good passwords are too difficult to remember. The human brain isn't wired to support that sort of data retention. However, if you care enough about the problem, you can use my solution of Keyring for PalmOS, an "electronic wallet" for security tokens that stores them all in a 3DES-encrypted database, unlockable with a single password, so you need remember only that one.

I would add to Wichert's recommendations:

The debian.org admins, as it turned out, didn't strictly need an IDS to know their machines had been compromised: They noticed the suspicious pattern of kernel "oopses", did a small amount of checking, and immediately drew the right conclusion. The nightly report from AIDE served mainly to confirm what they already knew. In general, an alert sysadmin is by far your best protection.

Security in general is a tough problem. Screw-ups and people shooting you in the foot are endemic, and meaningful improvement comes at a cost in inconvenience. I've barely scratched the surface of threat models that should be of concern — and there are other checking tools such as chkrootkit that are worth using. But I hope I've outlined some of the low-hanging fruit that yields the biggest improvements in areas that matter.

Resources:

Christophe Lupe's post about synergy between AIDE and Tripwire:
http://www.mail-archive.com/debian-security@lists.debian.org/msg11293.html

Moen's Laws and other lexicon items:
http://linuxmafia.com/~rick/lexicon.html

Wichert Akkerman's Debian.org Compromise 2003 pages:
http://www.wiggy.net/debian/developer-securing/

Nmap, a free open source utility for network exploration or security auditing:
http://www.insecure.org/nmap/
[2019 note: Now at https://nmap.org/.]

Snort, an open-source IDS of sorts (but networked, not host-based):
http://www.snort.org/

Logcheck, a script to detect anomalous logged events and mail the sysadmin:
http://alioth.debian.org/projects/logcheck/
[2019 note: Appears to have been revived at http://logcheck.org/.]

Tripwire, the original, classic host-based IDS. Notice that, although Tripwire is self-checking, it has the problem in common with all other host-based IDSes that intruders may disable or tamper it, to sabotage its protection. However, because every part of it, right down to the nightly reports, is cryptographically signed, it has the advantage of being extremely tamper-evident: If you ever fail to receive it nightly report, or get one that fails to validate as genuine, then you immediately know something's up. Having it check all files of your other IDS(es) further extends this advantage to those.
http://www.tripwire.org/
http://www.tripwire.com/

Tripwire note:
If running this verification regime on a suspect host strikes you as precarious, you're probably correct — and Tripwire, Inc. recommends that, at a minimum, you verify Tripwire files using the siggen utility provided for that purpose, and preferably store them on read-only media. Adjust to suit your level of paranoia (e.g., recompiling components using static linking, etc.).

AIDE, the younger challenger:
http://www.cs.tut.fi/~rammer/aide.html
[2019 update: Now at https://aide.github.io/.]

Integrit, a similar newcomer:
http://integrit.sourceforge.net/

Samhain, a similar newcomer that's said to be exceptionally good. A truly careful admin would run two lightweight IDSes, such as AIDE and Samhain, and have Tripwire check them both, in order to avoid having one IDS's flaws be a single point of failure:
http://la-samhna.de/samhain/

Prelude-IDS, another newcomer:
http://www.prelude-ids.org/
[2019 note: Has now evolved into Prelude SIEM, https://www.prelude-siem.com/en/.]

OPIE (One-time Password In Everything) and OpenSSH, via pam_opie module:
http://www.tho.org/~andy/pam-opie.html
http://www.derkeiler.com/Mailing-Lists/securityfocus/Secure_Shell/2003-02/0122.html
http://www.derkeiler.com/Mailing-Lists/securityfocus/Secure_Shell/2003-02/0121.html

S/Key and OpenSSH:
http://dbforums.com/arch/181/2003/6/823985
As with OPIE, you may need to recompile OpenSSH to ensure support:
http://www.sunfreeware.com/INSTALL.openssh

PalmKey:
http://palmkey.sourceforge.net/

Strip:
http://www.zetetic.net/products.html
[2019 note: Has now evolved into replacement codebase Codebook, https://www.zetetic.net/codebook/.]

pilOTP (proprietary):
http://astro.uchicago.edu/home/web/valdes/pilot/pilOTP/

Keyring for PalmOS:
http://gnukeyring.sourceforge.net/

Chkrootkit examines your system for common, known software toolkits used to conceal an intruder's presence after break-in ("rootkits"). As such, it gives only negative reassurance of "No, I don't see any of the signs I believe indicative of rootkits my designer taught me to look for", and in that sense is similar to a virus checker. Inherently, it cannot actually rule out the presence of rootkits it doesn't know about, let alone the intruders themselves.
http://www.chkrootkit.org/

Jim Dennis's Security Tips page has many further ideas:
http://www.starshine.org/sysadmoin/LinuxSecurityTips

Linuxmafia Knowledgebase (my PerlHoo documentation tree) also has further resources:
http://linuxmafia.com/kb/Security

 

Rick is [2010: was] a member of The Answer Gang.

[BIO] Rick has run freely-redistributable Unixen since 1992, having been roped in by first 386BSD, then Linux. Having found that either one sucked less, he blew away his last non-Unix box (OS/2 Warp) in 1996. He specialises in clue acquisition and delivery (documentation & training), system administration, security, WAN/LAN design and administration, and support. He helped plan the LINC Expo (which evolved into the first LinuxWorld Conference and Expo, in San Jose), Windows Refund Day, and several other rabble-rousing Linux community events in the San Francisco Bay Area. He's written and edited for IDG/LinuxWorld, SSC, and the USENIX Association; and spoken at LinuxWorld Conference and Expo and numerous user groups.

His first computer was his dad's slide rule, followed by visitor access to a card-walloping IBM mainframe at Stanford (1969). A glutton for punishment, he then moved on (during high school, 1970s) to early HP timeshared systems, People's Computer Company's PDP8s, and various of those they'll-never-fly-Orville microcomputers at the storied Homebrew Computer Club -- then more Big Blue computing horrors at college alleviated by bits of primeval BSD during UC Berkeley summer sessions, and so on. He's thus better qualified than most, to know just how much better off we are now.

When not playing Silicon Valley dot-com roulette, he enjoys long-distance bicycling, helping run science fiction conventions, and concentrating on becoming an uncarved block.

Copyright © 2004, Rick Moen, rick@linuxmafia.com.

This article was first published in issue 98 of Linux Gazette, January 2004.