By Rick Moen
Some weeks ago, I was spending some time assisting my mother-in-law, who's working on her Ph.D in computer network security, do a survey of a half-dozen or so Linux users at local user group, CABAL, about our security practices — with results like these:
Do we use anti-viral software? (No, except where we handle files or mail destined for Microsoft-OS machines.) Do we study our networks' security exposure using vulnerability-scanning software such as nmap and snort? (Many of us do, yes.) Do we run log-analysis security utilities such as logcheck? (Ditto.) Do we run Intrusion Detection System (IDS) suites such as Tripwire? (Almost never.) What measures do we take to eliminate security holes as they arise in a timely fashion? (Various.) Do we use kernel-level IP-traffic filtering scripts ("firewalls")? (Some do. Several more-experienced users operating all-Unix networks do not.)
These questions kept haunting me as I answered questions from new Linux administrators. Sometimes, those were very astute: Q: "How can you be sure that your system hasn't been compromised by hostile parties?" A: "Excellent question. You can't know, absolutely. A truly subtle and competent intruder manifests those abilities in part by being difficult to spot, and covering his or her tracks. But intruders (or their automated attack tools) generally break into a system to do some significant unauthorised activity, leaving clues that will be spotted by alert and capable admins who know their systems well enough to notice peculiar goings-on." That answer tends to leave questioners slightly uneasy — as intended.
I always told new admins that there two levels of threat to systems: from outside and from inside. Conventional thinking worries mostly about the former, e.g., the sort of perimeter survey you get by running
nmap -vv -sT -sR -O -n -oN tcpscan.log 10.0.1.3 nmap -vv -sU -sR -O -n -oN udpscan.log 10.0.1.3 nmap -vv -sA -sR -O -n -oN ackscan.log 10.0.1.3
...from the far side of your LAN (a different machine) against your IP=10.0.1.3 Linux host, to guesstimate how hard the latter's (figurative) exterior shell is — in the event of attack from elsewhere.
But the latter sort of threat — from the inside — is both more worrisome and more interesting. That is, if there's a entrance method (ssh, whatever) from the outside into your machine, and someone steals the password or other token required to use it, then you have unwelcome guests, who can then subvert your system's security from its own command prompt. It's well known that the latter step's often successful, but the real news is how easily and frequently passwords get stolen.
Consider inbound ssh access to your machine(s), a convenience much used and cherished by Unix users. Do you ever ssh in from machine you don't personally administer and have absolute confidence in? Even if you don't, do you always carry your ~/.ssh/known_hosts2 file with you, so you can be sure the remote host you reach is really yours? If you're ever lax on any of those matters, and you're even slightly unlucky, the bad guys will steal your access tokens and enter masquerading as you, later.
Even if you never do any of those things (making you a rare paranoic, indeed), can you say the same of all the friends you gave shell accounts to? Nobody ever used a cybercafe or university computer, or used PuTTY on a family Windows box teeming with spyware (or maybe even a keystroke-recording dongle connected to the keyboard)? Thought not. And there's your problem.
Which brings us to November's security incidents. A timeline should help us set the scene:
2003-08-25: Release of kernel v. 2.4.22 with an undetected memory-handling bug.
September 2003: Andrew Morton discovers that no bounds checking was being applied in kernel code to memory addresses passed to the brk() system call. Neither he nor anyone else posting to LKML is aware of the bug's security implications. However, an unknown bad guy, reading the Changelogs, realises those implications some time between this date and 2003-11-02.
2003-09-24: Andrew Morton commits a patch for the 2.6 kernel series.
2003-10-02: Marcelo Tosatti commits a patch for upcoming v. 2.4.23.
2003-10-09: Fix for brk() bug becomes available in 2.4.23-pre7 snapshot.
2003-11-02: Unknown Bad Guy breaks into FSF's savannah.gnu.org development host. Method of compromise is later claimed to be the same as those of the other machines mentioned below.
2003-11-19: Unknown Bad Guy exploits the bug to perform local-user root compromise of Debian Project development server named "master". From there, he/she compromises development servers klecker, murphy, and gluck. At no point were the Debian package archives compromised.
2003-11-20: Within one day, the Debian Project detects the compromise and shuts down all four machines for forensics and rebuild: Admins notice a suspicious pattern of kernel "oopses" and confirm their suspicions through being advised by AIDE (an IDS) on klecker, murphy, and gluck of unauthorised changes to /sbin/init and /usr/lib/locale/en_US.
2003-11-28: Release of kernel v. 2.4.23, incorporating the brk() fix.
2003-12-01: FSF discovers compromise of savannah.gnu.org .
2003-12-02: A Gentoo Project server (operated by a third party) participating in the rsync.gentoo.org cluster is compromised in what is claimed to be the same manner. Compromise is detected one hour later by an IDS and a file-integrity checker. No portage tree files (Gentoo software "packages") are compromised.
You'll notice how quickly the Debian and Gentoo people realised their problem, and corrected it — a point I'll come back to. But the first point to note is how the bad guy entered to begin with — a necessary first step before he/she could use the kernel flaw.
It turns out that one of the 1000+ Debian developers had been the victim of security-token theft. He or she used an ssh client on some machine, somewhere, that happened to have already been subverted. The ssh software was "trojaned" and privately logged his/her Debian-server login credentials, later conveying those to the attacker — who was then able to waltz in as if he/she were the developer. Only then did he or she use the kernel bug to escalate privilege to root-user access, something he/she might equally have done by finding an un-patched flaw in any other piece of security-sensitive software. The main point is: The intruder got in despite everyone (probably) being reasonably cautious and prudent, because one of his or her password-grabbing processes got lucky somewhere.
Two things put an immediate halt to this malarkey at the debian.org and gentoo.org sites. One was the presence of alert sysadmins, who, in debian.org's case, noticed the pattern of kernel "oopses" on two machines simultaneously, judged that far too great a coincidence, and thus were tipped off. (Similar alarm bells probably went through the gentoo.org admins' heads, but far fewer incident details have emerged from them.)
The other was a much-lauded but little-used type of software called host-based Intrusion Detection Systems, the classic example of which is Tripwire, invented by Gene Spafford and Eugene Kim at the fabled COAST security laboratory at Purdue University from 1992 through 1994. For most of its history, it was proprietary (with source code available for inspection but no right to independently develop it), offered for sale to business, and with a "free for non-commercial use" edition called Tripwire Academic Source Release (ASR) available for download.
Over time, Tripwire underwent a complete rewrite that unfortunately did nothing about the program's nagging usability issues (about which, more below), and then, under pressure from open-source alternatives and with help from VA Linux Systems, its sponsoring firm (Tripwire, Inc.) re-released Tripwire in October 2000 as open source software under the GNU GPL.
CTO Eugene Kim has, since then, professed indignation at the sparse participation in response by open source community coders — but the firm's pride and joy turns out to be non-portable C++ with no autoconf support. (Gosh, Gene, maybe those antediluvian coding standards, bizarre choice of language, and your firm's turning to open source only after mindshare had already fled to more-open alternatives have something to do with it?)
I remember Tripwire ASR; like most sysadmins, not at all fondly — having attempted to start living with it in 1994 and deciding it wasn't worth the hassle. It was and is an absolute horror to set up. In theory, you write a description of what files and directories (and what aspects of them) to check for unauthorised changes, have it take a snapshot of the current, non-compromised system state, and commit to disk all that information in a cryptographically verifiable state. Nice theory; teeth-grating execution.
Unfortunately, the tools and configuration syntax are impenetrably obscure, every operation runs incredibly slowly, and its system-integrity-checking mode churns out long and mostly meaningless reports to the root user, which must be studied and then used to further refine Tripwire's human-hostile ruleset to gradually refocus its attention on system changes that actually matter and cease reporting trivia. Because of the heavy use of encryption, each of those steps tends to be dog-slow, and the process must be run through iteratively, many times, using expert knowledge of one's system, before the results start to be useful and not just verbose babble.
Information overload, horrific configuration language, slow and performance-sapping operation, twisted administrative interface... argh! Save me from this! I quit wrestling with it, within a week or two.
Not at all coincidentally, starting one year before Tripwire went open source, it started getting serious open-source competition, starting with AIDE, a package by Rami Lehti and Pablo Virolainen in Finland. AIDE has lately been joined by similar designs starting in 2001: Ed L. Kashin's Integrit, Rainer Wichmann's Samhain, Yoann Vandoorselaere's Prelude IDS, and no doubt others.
I considered using AIDE, when it emerged in August 1999, and played with it a bit. Where Tripwire was slow and system-clogging, AIDE was fast and light. Where Tripwire was obscure and prone to breakage with puzzling errors, AIDE was easy to understand and debug. It had one big problem: The system-snapshot database, program binary, and configuration file weren't stored with cryptographic verification (as Tripwire does). The authors urged, instead, that those all be stored on write-protected media and updated only as needed.
Keeping the AIDE files on floppy or CDR is a major nuisance. The alternative, of just using them on the system's own hard drive, is easier but tends to give a false sense of security. That is, if/when the bad guy comes in and subverts your system, isn't he or she going to subvert the IDS, too? So, when an IDS tells you all is well, how do you know the bad guy isn't pulling its puppet strings? Tripwire has an answer to that objection; AIDE and friends do not.
That sort of false reassurance is the same one often encountered among users of RPM-based systems reassured by the results of running "rpm -Va" to "verify" the md5sum signatures of installed files: The values are "verified" against a simple Berkeley DB record in /var/lib/rpm — which of course a competent intruder will update to match his or her changes.
So, in the end, I didn't run AIDE routinely. The Debian Project developer boxes did, and it paid off — the intruder having been sophisticated enough to leverage a previously-unknown Linux kernel exploit, but not enough to notice AIDE and sandbag it before it could inform on him/her.
Following November's security incidents, and my mother-in-law's raised eyebrow over us Linux old-fogies not running host-based IDSes, I felt I had to revisit the matter, and explore options. To my great fortune, the last piece serendipitously arrived in a post to the debian-security mailing list by Lupe Christoph:
"We don't use AIDE exclusively at a client site, but in combination with Tripwire. We think Tripwire is a little more secure because it uses signed databases. So, we protect aide.db with Tripwire. AIDE is used for the parts Tripwire can't do because of its limited configurability...."
Um... yeah. Why didn't I think of that? Whacking Tripwire's configuration down to just the few minimum items it's best suited to handle, including AIDE's own otherwise-unchecked files, means the usual pain of using Tripwire fades into background noise, and makes its operations run in less than geologic time. Meanwhile, AIDE picks up the rest — and I don't have to worry that I'm fooling myself into complacency like an overconfident rpm user. It works a treat.
My Web site's "Lexicon" page includes Moen's Laws, such as:
Moen's First Law of Security: "It's easier to break in from the inside." E.g., many Internet break-ins result from masquerading as a legitimate user to gain user-level access, e.g., with sniffed passwords. The attacker then has a dramatically wider field of system weak points he or she can attack, compared to penetrating the system from outside.
Moen's Second Law of Security: "A system can be only as secure as the dumbest action it permits its dumbest user to perform." Your users are often your weakest link; smart bad guys will attack you there (e.g., via social engineering). Smart admins will try to compensate for this tendency, e.g., by using multi-factor authentication instead of just passwords.
Between the two of those, one could have predicted the sort of small calamity that overcame the Debian, Gentoo, and Savannah projects in November. Given the considerable likelihood of security tokens being stolen, especially on machines used by many people, it's a wonder it didn't happen sooner. The small miracle of that was that two of the three detected and fixed the break-in immediately — courtesy of host-based IDSes.
Detection is great, and better than a kick in the head (or living in a fool's paradise), but what about prevention? One way is to run an ssh daemon on an additional, non-standard port (maybe 2222 instead of 22) that requires OPIE or S/Key one-time passwords instead of regular, stealable ssh authentication. More precisely, one-time passwords can certainly be stolen, but then are useless because they've already been used up by the authorised user.
One-time passwords are a nuisance to manage: You generate a password "seed" and convey them somehow to your user. He or she either carries around a printout in very small type of the resulting series of 500 or so one-time passwords, crossing them off as they're used up, or puts the seed in a PalmPilot and generates those passwords from it using PalmKey, Strip, or pilOTP for PalmOS.
I may not use such a setup every time I'm away from home and tempted to cut corners — nor require my users to — but it might be nice to have that option the next time I'm in a cybercafe or some malware-infested bank of public Windows machines at a trade show.
Limiting remote shell (or similar) access, both by others and by yourself
...especially when it's from machines of doubtful integrity and/or shared-resource machines
Avoiding thinking you're lucky and trusting an unverified host key
In other ways, avoid making the error of using ssh without ensuring control of both ends, and avoid trusting the network between them.
Carrying a copy of your ~/.ssh/known_hosts2 file with yourself, e.g., on a USB flash drive in your pocket, so you can know that the ssh connection home really is reaching your machine rather than Prof. Moriarty's man-in-the-middle impostor machine.
Wichert Akkerman's page of information on the Debian.org compromise includes some intriguing recommendations to add to that, including some behavioural ones:
not ever ssh'ing from one remote host to another
using unique keys and passphrases for each host
disabling ssh passwd access and using only keys [public/private keypairs]
restricting the list of hosts that are allowed to ssh to your systems
The first of these is interesting and subtle: How many times have you ssh'd to someone else's machine, and then scp'd a file back to yourself ("pushing" it back to yourself)? Well, don't do that. Instead, scp it in "pull" mode from your own machine's command line:
$ scp username@remotehost:/tmp/somefile .
...rather than this form on remotehost's command line:
$ scp /tmp/somefile username@myhost:
Why? This gets back to the problem of stolen tokens, again: When you initiate the scp from remotehost to "push" the file back to where you are on myhost, you have to provide a stealable security token on a machine you don't control and have no reason to trust. "Pulling" the file from myhost poses no such risk.
Hardly anyone follows Wichert's second recommendation (unique passwords) because good passwords are too difficult to remember. The human brain isn't wired to support that sort of data retention. However, if you care enough about the problem, you can use my solution of Keyring for PalmOS, an "electronic wallet" for security tokens that stores them all in a 3DES-encrypted database, unlockable with a single password, so you need remember only that one.
I would add to Wichert's recommendations:
Pay attention, and know your systems well.
The debian.org admins, as it turned out, didn't strictly need an IDS to know their machines had been compromised: They noticed the suspicious pattern of kernel "oopses", did a small amount of checking, and immediately drew the right conclusion. The nightly report from AIDE served mainly to confirm what they already knew. In general, an alert sysadmin is by far your best protection.
Security in general is a tough problem. Screw-ups and people shooting you in the foot are endemic, and meaningful improvement comes at a cost in inconvenience. I've barely scratched the surface of threat models that should be of concern — and there are other checking tools such as chkrootkit that are worth using. But I hope I've outlined some of the low-hanging fruit that yields the biggest improvements in areas that matter.
Christophe Lupe's post about synergy between AIDE and
Moen's Laws and other lexicon items:
Wichert Akkerman's Debian.org Compromise 2003 pages:
Nmap, a free open source utility for network exploration or
[2019 note: Now at https://nmap.org/.]
Snort, an open-source IDS of sorts (but networked, not
Logcheck, a script to detect anomalous logged events and
mail the sysadmin:
[2019 note: Appears to have been revived at http://logcheck.org/.]
Tripwire, the original, classic host-based IDS. Notice that,
although Tripwire is self-checking, it has the problem in
common with all other host-based IDSes that intruders may
disable or tamper it, to sabotage its protection. However,
because every part of it, right down to the nightly reports, is
cryptographically signed, it has the advantage of being
extremely tamper-evident: If you ever fail to receive it
nightly report, or get one that fails to validate as genuine,
then you immediately know something's up. Having it check all
files of your other IDS(es) further extends this advantage to
If running this verification regime on a suspect host strikes you as precarious, you're probably correct — and Tripwire, Inc. recommends that, at a minimum, you verify Tripwire files using the siggen utility provided for that purpose, and preferably store them on read-only media. Adjust to suit your level of paranoia (e.g., recompiling components using static linking, etc.).
AIDE, the younger challenger:
[2019 update: Now at https://aide.github.io/.]
Integrit, a similar newcomer:
Samhain, a similar newcomer that's said to be exceptionally
good. A truly careful admin would run two lightweight IDSes,
such as AIDE and Samhain, and have Tripwire check them both, in
order to avoid having one IDS's flaws be a single point of
Prelude-IDS, another newcomer:
[2019 note: Has now evolved into Prelude SIEM, https://www.prelude-siem.com/en/.]/p>
OPIE (One-time Password In Everything) and OpenSSH, via
S/Key and OpenSSH:
As with OPIE, you may need to recompile OpenSSH to ensure support:
[2019 note: Has now evolved into replacement codebase Codebook, https://www.zetetic.net/codebook/.]
Keyring for PalmOS:
Chkrootkit examines your system for common, known software
toolkits used to conceal an intruder's presence after break-in
("rootkits"). As such, it gives only negative reassurance of
"No, I don't see any of the signs I believe indicative of
rootkits my designer taught me to look for", and in that sense
is similar to a virus checker. Inherently, it cannot actually
rule out the presence of rootkits it doesn't know about, let
alone the intruders themselves.
Jim Dennis's Security Tips page has many further ideas:
Linuxmafia Knowledgebase (my PerlHoo documentation tree)
also has further resources:
Rick is [2010: was] a member of The Answer Gang.
Rick has run freely-redistributable Unixen since 1992, having been roped
in by first 386BSD, then Linux. Having found that either one
sucked less, he blew
away his last non-Unix box (OS/2 Warp) in 1996. He specialises in clue
acquisition and delivery (documentation & training), system
administration, security, WAN/LAN design and administration, and
support. He helped plan the LINC Expo (which evolved into the first
LinuxWorld Conference and Expo, in San Jose), Windows Refund Day, and
several other rabble-rousing Linux community events in the San Francisco
Bay Area. He's written and edited for IDG/LinuxWorld, SSC, and the
USENIX Association; and spoken at LinuxWorld Conference and Expo and
numerous user groups.
His first computer was his dad's slide rule, followed by visitor access
to a card-walloping IBM mainframe at Stanford (1969). A glutton for
punishment, he then moved on (during high school, 1970s) to early HP
timeshared systems, People's Computer Company's PDP8s, and various
of those they'll-never-fly-Orville microcomputers at the storied
Homebrew Computer Club -- then more Big Blue computing horrors at
college alleviated by bits of primeval BSD during UC Berkeley summer
sessions, and so on. He's thus better qualified than most, to know just
how much better off we are now.
When not playing Silicon Valley dot-com roulette, he enjoys
long-distance bicycling, helping run science fiction conventions, and
concentrating on becoming an uncarved block.
His first computer was his dad's slide rule, followed by visitor access to a card-walloping IBM mainframe at Stanford (1969). A glutton for punishment, he then moved on (during high school, 1970s) to early HP timeshared systems, People's Computer Company's PDP8s, and various of those they'll-never-fly-Orville microcomputers at the storied Homebrew Computer Club -- then more Big Blue computing horrors at college alleviated by bits of primeval BSD during UC Berkeley summer sessions, and so on. He's thus better qualified than most, to know just how much better off we are now.
When not playing Silicon Valley dot-com roulette, he enjoys long-distance bicycling, helping run science fiction conventions, and concentrating on becoming an uncarved block.