[sf-lug] interlopers continued

[Michael Paoli]

[snipped: lots of stuff about "interesting" looking apparent IRC
connections, e.g.:
http://linuxmafia.com/pipermail/sf-lug/2007q3/001707.html
et. seq.]

If the host is *not* root compromised, one may want to investigate
the connections a bit more, to determine if they're legitimate,
or case of authorized user exceeding their authority / violating policy
(but short of root compromise) ... or maybe some (quasi-?)legitimate
software with perhaps some undesired behavior(s).

E.g. with fuser or lsof (and possibly other tools too),
one should be able to isolate those TCP connections to the specific PIDs
that have them open.  From there, one can use ps and the /proc filesystem
to find out a lot more about those PIDs (e.g. what binary are they running,
what command line arguments/options, and what UID (user/login) is using
them, when did they start, from what directory, what's the PPID, TTY, etc.

Sometimes stuff that looks funky/suspicious ends up having a fairly easy
(but sometimes surprising) explanation.  It's often observed that when
folks set up firewalls (and monitoring/logging thereof) and/or look at
their various network activity in sufficient detail, especially if they're
not rather to quite well acquainted with such traffic, they'll think lots
of stuff they see is an "attack" or security problem or the like, when
it isn't really.  Of course there may also be illegitimate traffic,
problems, "attacks" or such in there too (speaking of which, I just blocked
another ssh "dictionary" attack as I'm typing this).

So, ... while it's good to at least keep in the back of your head that
stuff one sees may be indicative of a problem - or perhaps the system has
already been root compromised, ... it's often useful to first look for
more legitimate explanations ... and follow through on either or both
hypotheses until one can have a suitably high degree of confidence in
precisely one of those hypotheses.

Miscellaneous bits of related technical information.
UNIX/LINUX manual pages, typically noted in form, e.g.:
man(1)
which would refer to the manual page on man, in section 1 of the UNIX/LINUX
manual.
This manual page can be examined via the man command, e.g. for fuser(1)
man 1 fuser
The section information can be omitted if the command is first found
in man page ordering in the desired section or the command only exists
in that one section.  Precise syntax for specifying man sections varies
among UNIX/LINUX/BSD flavors.
*nix (or *NIX) if often used as a shorthand for UNIX-like families of
operating systems (most notably LINUX, UNIX, and BSD, although there are
some others too).
E.g. using fuser and friends to investigate a connection:
$ netstat -an | fgrep 208.96.15.254:22
tcp        0      0 198.144.194.236:36594   208.96.15.254:22        ESTABLISHED
$ fuser -n tcp 36594
36594/tcp:           23160
$ ps lwwwww 23160
F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME COMMAND
0  1003 23160 23116   9   0  5124  392 select S+   pts/12     0:01 ssh -ax
new.balug.org.
$ ps uwwwww 23160
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
michael  23160  0.0  0.0  5124  392 pts/12   S+   Jun20   0:01 ssh -ax
new.balug.org.
netstat(8)
fuser(1)
ps(1)
lsof(8)
PID - Process ID
PPID - Parent Process ID
UID - User ID
GID - Group ID
{E,R}{U,G}ID - {Effective,Real}{User,Group}ID
fork(2)
exit(3)
wait(2)
I/O - Input/Output
set[r]e{u,g}id(2)
{a,b} - syntax often used (like C-shell) to show alternatives - either a or b
[foo] - man page like syntax also used to show optional portions
[abc] - Bourne/Bash/POSIX etc. syntax often used to show match to any one
        of the characters within the square brackets (see shell man pages
        for some of the more specific details/exceptions)
context matters, e.g. what is meant by [abe] will depend on context

If a system is root compromised, one cannot rely upon what it claims to
be the case - e.g. the output, behavior and actions of all commands are
suspect and may not be telling the truth at all.

If the host showing IRC connections was in fact root compromised, it's fairly
likely that netstat would be replaced, and that the illegitimate replacement
netstat command would likely not show the illegitimate connections.
Of course root compromises vary in their thoroughness of hiding the
compromise.  Perhaps some funky utility or such makes use of those IRC
connections?

Also, tools such as tcpdump may be useful to see what's being done with
those apparent IRC connections.  If the host is in fact root compromised,
that data could still be examined elsewhere on the network by a host
that hadn't been compromised (e.g. a LINUX based router with
tcpdump available on it).

Watching for unusual patters - e.g. in network traffic and/or disk I/O - is
often useful in detecting or noticing evidence of compromise, unauthorized
traffic/connections, or other problems (program gone nuts).  E.g. is one's
network activity LED showing heavy continuous activity when one would
expect it to have little to no activity?

And thus far still the case: how are most root/Administrator compromises
"detected"?  Not by folks that are (or should be) administering those
compromised systems, but by systems/network administrators of other systems
who complain that: "Hey, *your* box is beating away trying to
attack / break into *my* box/network!"