From: Rick Moen rick@linuxmafia.com
Newsgroups: linux.astcomm.net
Subject: Re: More on being cracked
Date: Sun, 8 Sep 2002 20:17:02 +0000 (UTC)
Organization: If you lived here, you'd be $HOME already.
User-Agent: tin/1.5.13-20020703 ("Chop Suey!") (UNIX) (Linux/2.2.19 (i686))

Chris Olson chris@astcomm.net wrote:

> Could be. I want to know how it got there.

A worthy goal. As I was saying, computer forensics is a large field of
study, on its own.

The first steps in formal forensics would be as follows:

(1) Immediately yank AC power to the compromised host upon determining
it to be compromised, shutting it off summarily without going through
shutdown. There are reasons for this, which we could go through but
I don't want to spend time on in this newsgroup article.

(2) Inform your management (if any). Inform law enforcement, if you
think you might wish to pursue any form of criminal investigation.
Law enforcement agencies may have particular requirements as to chain
of custody on evidence, and other matters, that they might need you to
comply with.

(3) Secure copies of logs from all other relevant hosts.

(4) Boot other media (a separate hard drive, an LNX-BBC disc, whatever
works for you), and create a bitwise copy of the compromised host's
filesystems. Do this in such a fashion that you avoid executing any
code from the compromised filesystems. Shut down, extract the
compromised system's original hard drive(s) as a reference copy, and
store for safekeeping.

(5) Boot up again, still using other boot media to avoid executing any
code from the replica of the compromised filesystems. Mount them.
Examine at your leisure (along with the preserved logfiles from
elsewhere), to attempt to find the entry point by tracing any
still-available signs of the intruder's actions.

(6) In parallel, you would of course also need to construct and deploy a
replacement host. You can safely reuse without particular scrutiny all
data files. (Safe from those files posing a renewed security threat, in
themselves. The intruder could of course have corrupted data files,
substituted fake files, etc.) You cannot reuse, but need to refer to,
the prior contents of /etc. You cannot reuse executables including
interpreted code and script files.

The CERT document I referred you to earlier,
http://www.cert.org/tech_tips/win-UNIX-system_compromise.html, has more.
(It's one of the few useful things CERT does.)

> After reviewing firewall logs from the log server, we've ruled out
> network intrusion and direct access from the outside.

Did you ever ssh or scp into the host from any other host? Then, please see
http://linuxmafia.com/~rick/linux-info/breakin-without-remote-vulnerability .

I believe you said the host ran an sshd. Crypto, alas, is not magic
security pixie dust. SSH tunnels' security hinges on (1) authentication
of the remote host (to rule out man-in-the-middle attacks) and (2) the
assumption that _neither_ host has been compromised. Therefore, if you
ssh in from a compromised host, you lose.

(Point #1 refers to the problem of transporting host keys out of band,
in advance of the first SSH session. Most people just shrug off this
risk, and live with the possibility of MitM attacks.)

> Since I didn't deliberately download a rootkit/trojan tarball and
> install it....

Just for clarity: A rootkit is _not_ an attack tool. A rootkit is a
set of trojaned replacements for standard system utilities that the
intruder installs _after_ breakin, to conceal his presence as long as
possible.

Symlinking /root/.bash_history to /dev/null is an additional, incredibly
sloppy way to further that goal. Ditto erasing logfiles, disabling
syslogd, etc. (I say "incredibly sloppy" because it's a dead giveaway.)

If I may immodestly suggest it, again:
http://linuxmafia.com/~rick/essays/attacking-linux.html

> Call it "unfocussed, randomly flailing paranoia" if you will, but until
> I know for sure where it came from, only then will I be able to
> determine if the same infected software has been installed on other
> machines here.

Suggestion: Since your working hypothesis is "infected software", find
it. If you fail to find it, that would tend to suggest the hypothesis
failed. (Personally, I very strongly doubt the hypothesis, ab initio.
But I don't think you're going to get very far if -- no offence -- you
didn't even know what a rootkit is and isn't.)

> Until that time of enlightenment, every source is suspect, including
> Debian's software vault, the 'test' packages for KDE that were
> installed on this machine, and several other sources.

Consider ramifications: If a major upstream repository started handing
out security-compromised software a week ago, odds are that it would be
all over the news by now. In this regard, unofficial apt repositories
always entail more inherent risk than official ones. Pick your poison.
But I'll lay very long odds against "boojums" from the two sources
you've cited specifically, on the evidential grounds just described.

Suggestions for further reading, after you recover from what was
undoubtedly a very troublesome and exhausting experience:

Crypto-gram. Comes out the 15th of each month. Essential.
http://www.counterpane.com/crypto-gram.html

The RISKS Digest. Irregularly issued. Essential. Funny as hell.
http://catless.ncl.ac.uk/Risks/

Firewalls and Internet Security, Wm. R. Cheswick and Steven M. Bellovin
Addison-Wesley, ISBN 0-201-63357-4
Steve tells me that the 2nd edition is due out Real Soon Now.

Real World Linux Security, by Bob Toxen
Prentice Hall, ISBN 0-13-028187-5
Generally good, though Toxen includes some dumb-ass recommendations that
actually drove me to offer to Bill Pollock of No Starch Press to write a
Linux security book of my own -- which I may do, if I can find an angle
that will distinguish mine from the crowd.

Running Linux, by Matt Welsh et alii.
O'Reilly, 3rd edition. ISBN 1-56592-469-X
A prerequisite for any hope of security is to understand your systems
and their environment. This is one place to start.

TCP/IP Network Administration, by Craig Hunt
O'Reilly, 3rd edition. ISBN 0-596-00297-1
And this is the other.

--
Cheers, "That article and its poster have been cancelled."
Rick Moen -- David B. O'Donnel, sysadmin for America Online
rick@linuxmafia.com