[sf-lug] how to whack crackers

[Rick Moen]

Quoting jim (jim at well.com):

>    hoping for suggestions to defend against hackers: 

Hmm, I think you've already somewhat gone amiss with your (implied)
mental model:  In general, taking an already-built *ix system as a
given, fixed thing and then asking how you can "defend" it is a losing
strategy.  Instead, you should ask yourself what threats against the
machine are worth worrying about, and how to do some combination of the
following concerning them:  prevention, detection, damage reduction,
defence in depth, hardening, identification of the attackers, and
recovery.  A big part of that should be re-examining the basic contents
and design of the entire system.  But you have to start with assessing
and understanding threat models, a point I'll return to, below.

Some guy I shave once wrote in an IDG article
(http://www.itworld.com/print/36437):

   Ozancin's talk was fairly comprehensive and typical of such security
   talks in its emphasis: it focused almost entirely on prevention and
   detection.

   Prevention and detection are, of course, very good things, but
   ideally they should be part of a better-rounded effort at risk
   assessment and management. That should include damage reduction (what
   is at risk?), defence in depth (how can we avoid having all our eggs
   in one basket?), hardening (e.g., jumpering the SCSI drives read-only
   for some filesystems, and altering Ethernet hardware to make
   promiscuous mode impossible), identification of the attackers, and
   recovery from security incidents. Explicit security policies,
   security auditing, the design and testing of backup systems, automatic
   and manual log analysis, handling of dialup access, physical security
   for the network, the special problems posed by laptop users, security
   training and documentation, and disaster recovery and costing are
   necessary parts of such an effort.



>    we've got a box on the internet using a speakeasy 
> IP address. a linksys home router sees the front end 
> and NATs traffic for ssh and http to the box, which 
> is a node on the LAN running ubuntu server 8.10. 
>    crackers regularly knock on the door. we've 
> implemented IP tables, though they don't work as we 
> think they should. for example: 
> 
> we have a rule (one of many similar) 
> -A INPUT -p tcp -m iprange \
> --src-range 110.0.0.0-126.255.255.255 \
> --dport 22 -j DROP 
> 
> iptables -L shows 
> DROP  tcp  --  anywhere  anywhere  source IP range \
> 110.0.0.0-126.255.255.255 tcp dpt:ssh 
> 
> and yet /var/log/auth.log shows ssh login attempts 
> for a variety of user names from 
> 124.93.200.34 

You're impliedly classifying this as a problem.  I question whether it's
actually a threat in any meaningful sense, even though:  Any box exposed
to Internet traffic on any port is going to get probed, many, many times
a day.  As part of that, anyone running sshd will get the Internet
equivalent of twisting the doorknob, which is basically what you've
described above, many, many times each and every day.

For analogy's sake, let's say you visit a nice city and stay in a hotel.
You go out, but carelessly leave your room door unlocked.  When you
return, you've been burglarised.  

Dismaying, to be sure.  However, is it appropriate to go shopping for
complex door-management hardware?  Wouldn't it be smarter, understanding
as you do the threat model of people twisting hotel doorknobs and
burglarising rooms that have been carelessly left unlocked, to just not
leave your door unlocked in the future?

In the case of *ix systems, there are system cracklibs, i.e., the system
can and should insist that people use strong passwords whenever they run
/usr/bin/passwd.[1]  the /etc/pam.d/common-password PAM configuration
file controls this.  You probably have something like this, there:


  # /etc/pam.d/common-password - password-related modules common to all
  # services
  #
  # This file is included from other service-specific PAM config files,
  # and should contain a list of modules that define  the services to be
  # used to change user passwords.  The default is pam_unix

  # The "nullok" option allows users to change an empty password, else
  # empty passwords are treated as locked accounts.
  #
  # (Add `md5' after the module name to enable MD5 passwords)
  #
  # The "obscure" option replaces the old `OBSCURE_CHECKS_ENAB' option in
  # login.defs. Also the "min" and "max" options enforce the length of the
  # new password.

  password   required   pam_unix.so nullok obscure min=4 max=8 md5

  # Alternate strength checking for password. Note that this
  # requires the libpam-cracklib package to be installed.
  # You will need to comment out the password line above and
  # uncomment the next two in order to use this.
  # (Replaces the `OBSCURE_CHECKS_ENAB', `CRACKLIB_DICTPATH')
  #
  # password required       pam_cracklib.so retry=3 minlen=6 difok=3
  # password required       pam_unix.so use_authtok nullok md5

As the system says, you probably want to install the libpam-cracklib
package and uncomment the two lines in the above file, to enable it.
This page has full details:
http://www.deer-run.com/~hal/sysadmin/pam_cracklib.html

>    the box has been cracked once already, we fixed 
> that vulnerability (i didn't think about a well-known 
> default user, ubuntu: someone guessed that user and 
> password, which was probably a well-known default). 

So, you did something very unwise and got caught napping.  Don't feel
too bad; in 1995, I (similarly) made the mistake of assuming the Debian
AWstats package has security-safe defaults, and got my Web front page
defaced.  At the time, I was so surprised and panicked that, in
response, I (incorrectly) assumed the server was root-compromised and
rebuilt it from scratch, unnecessarily.[2]  See:
http://linuxmafia.com/news.html

By the way, your above wording raises a couple of concerns:

1.  Does the term "cracked" in this context entail _root_ compromise?
I.e., did the intruder merely enter the system masquerading as a
legitimate user ("ubuntu")[3] whose password he/she guessed -- or, more
likely, whose automated script guessed that password -- or did the
intruder do that _and_ then escalate privilege to root-user level or
equivalent?

That, you see, is a really vital question.  If you don't know, you
really, really need to find out, before doing anything else.  If you
still cannot tell after examination, then the safest course of action is
to assume root compromise and do a ground-up system rebuild[4], which is
of course painful.  The experience of that level of pain can be, in
itself, instructive.  In 2005, it taught me to be skeptical about the
security of Web apps, for example.  ;->  (Eventually, it also taught me
a few pointers about how a set of system Web pages might be compromised
without system _root_ compromise, but at that point I'd already slightly
overreacted and done a full rebuild.  Erring on the side of pessimism in 
such matters is unfortunate but probably wiser than the opposite error.)

2.  If your system _did_ get root-compromised, does your phrase "fixed
the vulnerability" mean you did a ground-up system rebuild?  Because,
after root compromise, you really have no alternative but to shut the
system down, secure a copy of the datafiles only, grab an archive set
of the system configuration files and user dotfiles for reference only
but not re-use, rebuild software from trusted media, reconstruct system
configuration files manually referring to the archive set of old
conffiles, restore users' data but not their dotfile directories or
executables, set all new passwords, study the old configuration to
ensure that similar threats are not a problem, post a system bulletin
explaining what happened, and convey to all users their new passwords
out-of-band.

If you do anything less than that (following root compromise), you
haven't fixed the problem.

I stress the above because I've noticed that many newcomers to *ix
aren't taking that situation seriously.  They either don't even try to
determine, following a security incident, whether the system is now root
compromised, and just hope for the best, or seriously think that a
root-compromised system can be "fixed" through some means short of a
ground-up rebuild.

And, anyway, _who_ created "default user ubuntu"?  I really doubt that a
default Ubuntu install creates any such "joe account", as that would be
a ghastly design error.

I suspect a system-local human screwup, e.g., someone with admin
privilege deciding that it'd be a good idea to have login account
"ubuntu" with password "ubuntu".  

Basically, your enforced system policy should preclude creation of "joe
accounts".  Any.



>    ideas we have include 

If you'll pardon my airing an opinion based on long years observing such
discussions, most users' "ideas", and many programmers' ideas, about
security are bad ones.  

The root cause of that badness is failure to start with threat models:
That term refers to a tracing out of a particular potential threat to a
system as to mechanism, severity, what's at risk, credibility (is it
even worth worrying about?), and so on.  You cannot rationally start
taking measures until you understand what you're taking measures
against, and why, and how.

And, to understand threats against your system, you must first and
foremost understand your system.

Many users' "ideas" boil down ultimately to little more than technophile 
gadget-freakery, throwing additional software semi-randomly at poorly
comprehended threats, and thereby making the system more difficult to
understand and its behaviour more difficult to predict.  Many of those
"ideas" also, themselves, give rise to _new_ threats against the system
(often but not always denial of service threats).

Avoidable complexity is, in general, bad from several perspectives
including security.  And most users' instinct, when you ask them for
security suggestions, is to add additional layers of software, which of
course means greater complexity.


> * mount most filesystems in read-only mode (excepting 
> /var/log/, which is a separate mount point)

This is useful, but you should carefully consider why, e.g., what
purposes does it serve and not serve.  You also might, in a similar
sense, set some mount _options_ for specific parts of the file tree,
limiting what can be done with/to them.

Here's the partition map for the current, 11-year-old linuxmafia.com
box[4]:


# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nosuid          0       0
/dev/sda5       /               ext3    defaults,errors=remount-ro 0 1
/dev/sda1       /boot           ext2    nodev,nosuid,noexec,ro 0       2
/dev/sdb1       /home           ext3    nodev,nosuid    0       2
/dev/sdb5       /tmp            ext2    noatime,nodev,nosuid 0       2
/dev/sdb8       /usr            ext2    nodev,ro        0       2
/dev/sda9       /usr/local      ext3    defaults        0       2
/dev/sdb7       /var            ext3    noatime,nodev,nosuid 0       2
/dev/sda8       /var/log        ext2    noatime,nodev,nosuid 0       2
/dev/sda7       none            swap    sw              0       0
/dev/sdb6       none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0
/dev/sda6       /mnt/recovery   ext2    rw,noauto       0       2


# fdisk -l /dev/sda

Disk /dev/sda: 9105 MB, 9105024000 bytes
64 heads, 32 sectors/track, 8683 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          95       97264   83  Linux
/dev/sda2              96        7369     7448576    5  Extended
/dev/sda5              96        1049      976880   83  Linux
/dev/sda6            1050        1526      488432   83  Linux
/dev/sda7            1527        1647      123888   82  Linux swap / Solaris
/dev/sda8            1648        2601      976880   83  Linux
/dev/sda9            2602        7369     4882416   83  Linux


# fdisk -l /dev/sdb

Disk /dev/sdb: 9105 MB, 9105024000 bytes
64 heads, 32 sectors/track, 8683 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         954      976880   83  Linux
/dev/sdb2             955        6129     5299200    5  Extended
/dev/sdb5             955        1240      292848   83  Linux
/dev/sdb6            1241        1361      123888   82  Linux swap / Solaris
/dev/sdb7            1362        3268     1952752   83  Linux
/dev/sdb8            3269        6129     2929648   83  Linux


Now, some of that's a bit antiquated, but you might find some useful
ideas in there -- including noting what is _not_ done.  I found out
through trial and error, for example, that it's a bad idea to set the
noexec mount option for /tmp on an ongoing basis, because package
postinst scripts get run from there.  (Which is a pity, since otherwise
one would not want executables to run, there.)  I am tempted to set
noexec on /tmp, and create an "apt" hook to remount it temporarily to
allow the exec bit during package operations only, and then remount
afterwards.  (I already have such a hook for the /usr read-only mount.)

What purposes it does serve:  The read-only mounts protect against the
biggest single threat to your system, which is _you_, the sysadmin.
That is, I (and processes I choose to run) cannot accidentally clobber
/usr or /boot as the root user, because I must remount those filesystems
read-write before I can screw with them.

Also, there are performance advantage to some of the options and
techniques mentioned, e.g., jettisoning the atime stamp on filesystems
that don't need it, and eschewing the overhead of an ext3 journal on
filesystems that either are disposable (/var/log, /tmp) or are normally
read-only (/usr, /boot).

What purposes it does not serve:  A non-automated intruder with
root-level access would not be meaningfully impeded by any of the
partitioning and mount-option controls cited above, because, having root
access, he/she can remount anything as desired, trivially.

However, some of the read-only and restricted options cited above
_could_ very well sabotage many automated attacks if, e.g., in a moment
of weakness I deployed a vulnerable PHP app.  Consider, for example, the
Lupper worm of November 2005, as I described it in
http://linuxmafia.com/~rick/faq/index.php?page=virus#virus5 :  If
triggered remotely via one of the criminally vulnerable target Web
codebases, the remote attack used system calls to run wget, and fetch an
exploit binary to /tmp, and then run that executable.  Setting noexec on
/tmp (which I currently do not do, but am considering again) would
therefore prevent the exploit phase of that canned attack.



> * have sshd listen on some upper port rather than 22 
> (and change iptables rules accordingly) 

This is a very popular, classic bad idea:  It's one of the canonical
examples of "security through obscurity", and really gains you nothing
worth having, when seen in reasonable context.

linuxmafia.com's sshd has been on the Internet since the invention of
ssh, _on_ port 22 (along with several others, so that particular users
can have ingress despite stupid corporate firewalls on their end that
block their connection to outbound port 22), and gets doorknob-twisting 
checks for "joe accounts" from random locations on the Internet many
times each day.  No security compromise has ever resulted, nor would I
expect any, because I can do the math.

It's simply not possible to hit up my sshd with a rapid enough barrage
of guessed passwords even for known logins such as "rick", to have any
reasonable likelihood of success in a reasonable time span (like, years).
Even if it were, I'd notice my system falling over because /var/log/*
had overfilled, long before an attacker had even a tiny chance of
success through such a "dictionary attack" on my sshd.

Thus, with all respect to Prof. Verma and Tom Haddon, I deliberately
eschew not only running sshd only on non-standard ports but also things
like DenyHosts and Fail2ban, as pointless because they attempt to solve
the wrong problem.  The correct problem to solve is not "How do I
prevent continual doorknob-twisting?", but rather "How do I ensure that
doorknob-twisting doesn't work?"

It doesn't work on linuxmafia.com because the system has no "joe
accounts", and because dictionary attacks against an sshd statistically
have no prayer of working unless such accounts are present.

> * have a cron job run every five minutes to monitor 
> the box, mainly checking for weird user activity and 
> probably shutting down the box upon discovering such. 
> * /etc/hosts.deny has ALL=PARANOID and some ip addresses 
> that crackers have used on us. 

"Some IP addresses that crackers have used on us"?  C'mon, that's just
dumb.  You have no hope of keeping up, and they have an effectively
unlimited supply of IP addresses, given that many of them are using
zombified MS-Windows desktop boxes as attack vectors.

You really should consider spending the same amount of effort ensuring
that there's nothing on the local system that easily attackable, instead
of trying to wall off potential outside points of origin for attacks
against local vulnerabilities.


> we're not happy with our ideas as a complete defense and hope some
> of you will chime in with opinions about our ideas as well as ideas
> we haven't thought of. 

OK.  Start over.

The ideas you've started with are steps in the wrong direction.

Make sure you understand your system, its weak points, and its probable
avenues of attack.  Make sure you've addressed _all_ aspects of the
problem:  prevention, detection, damage reduction, defence in depth,
hardening, identification of the attackers, and recovery.

Some stuff from the guy I shave:

http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html
(You _do_ have a planned-out backup/restore routine, right?)

http://linuxgazette.net/issue98/moen.html
http://www.itworld.com/print/36437
http://linuxmafia.com/faq/Security/breakin-without-remote-vulnerability.html
http://linuxgazette.net/118/weatherwax.html#1


[1] Note that this is not effective against users wielding root
authority including root-equivalent privilege using sudo.

Also note that it doesn't preclude people using the same credentials on
your system as they do elsewhere, which can lead to user-level breakin 
via stealing of the credentials elsewhere.  Again, see
http://linuxmafia.com/faq/Security/breakin-without-remote-vulnerability.html
(***COUGH*** shells.sourceforge.net ***COUGH***).

[2] Rebuilding from scratch means that you assume no executable or lib
or configuration file or dotfile can any longer be trusted, because you
estimate that a hostile party has gained root-user access and must
assume he/she has subverted the system.  Everything else that's
necessary to the task of rebuilding follows from that assumption.

[3] There really should not have been such a user.  A *ix system should
not have "role" accounts:  You want individual accountability.  This
general rule does not, however, rule out per-system-process usernames
that the processes run as, such as postgres, bind, sshd, etc.  The
latter should generally have their login shells set as /bin/false in
/etc/passwd, so that they're not even theoretically possible as shell
users.

[4] Well, the hardware's 11 years old.  I moved the site to that
hardware around 6-7 years ago.