[sf-lug] how to whack crackers
jim
jim at well.com
Mon Jan 5 16:00:46 PST 2009
rick! i've been thinking of you rather a lot lately.
thanks for your reply. briefly:
we took the box out and put a new, well--different,
one in and rebuilt with ubuntu server 8.10. as The
Cracked Box was ubuntu server 7.10 and we'd only
started putting things on it, i didn't bother to
reclaim data. and anyway, doing it from scratch will
be good practice and won't have some of the stupid
stuff we'd had on The Cracked Box.
no, the cracker did not get root access (not that
it matters, as the box and all its files is sitting
"over there" in the corner).
i made sure there are no well-known user accounts
on The New Box and that as many as possible (e.g.
daemon accounts) have /bin/false or some such.
(i did make a user account that has /usr/bin/yes
as its "shell", and that works great! the screen of
the person logging in as that user gets repeated
lines with the 'y' character on it until they hit
CTL-C or some such. "fixed the vulnerability" ==
removed The Cracked Box and ensured there are no
similar accounts on The New Box.)
i cannot figure out a good backup scheme. the one
that copies absolutely everything from certain
directories each night is inelegant.
we recognize that the growing /var/log/auth.log file
represents doorknob tests, it's unnerving, possibly
educational. and the big number of iptables rules seems
to have no effect: maybe we've learned that lesson, too.
there's nothing much that's worth defending other than
the use of the box itself, but that's worth defending.
i've manually gone over many filesystems on many boxes
over the last years looking for directories with funny
names beginning with the '.' character and perhaps one
or more whitespace characters. it's a tedious job but
kind of suits me at times.
there are only three humans that may log into the box,
so password enforcement, for now, can be verbal abuse
and stern reminders. i'll have to get pam's cracklib
working.
the only threat model that i can make out is that some
cracker gets through the sshd door and does something:
* puts on a root kit
* puts in tools that support a bot net
* other (can't think of what)
we will have a web server running, and i'm sure i'll
have lots of new lessons to learn with that. got any httpd
threat models? (all threat model news is welcome.)
i will re-read and re-re-read your email and likely
have questions.
many thanks,
jim
On Mon, 2009-01-05 at 14:37 -0800, Rick Moen wrote:
> Quoting jim (jim at well.com):
>
> > hoping for suggestions to defend against hackers:
>
> Hmm, I think you've already somewhat gone amiss with your (implied)
> mental model: In general, taking an already-built *ix system as a
> given, fixed thing and then asking how you can "defend" it is a losing
> strategy. Instead, you should ask yourself what threats against the
> machine are worth worrying about, and how to do some combination of the
> following concerning them: prevention, detection, damage reduction,
> defence in depth, hardening, identification of the attackers, and
> recovery. A big part of that should be re-examining the basic contents
> and design of the entire system. But you have to start with assessing
> and understanding threat models, a point I'll return to, below.
>
> Some guy I shave once wrote in an IDG article
> (http://www.itworld.com/print/36437):
>
> Ozancin's talk was fairly comprehensive and typical of such security
> talks in its emphasis: it focused almost entirely on prevention and
> detection.
>
> Prevention and detection are, of course, very good things, but
> ideally they should be part of a better-rounded effort at risk
> assessment and management. That should include damage reduction (what
> is at risk?), defence in depth (how can we avoid having all our eggs
> in one basket?), hardening (e.g., jumpering the SCSI drives read-only
> for some filesystems, and altering Ethernet hardware to make
> promiscuous mode impossible), identification of the attackers, and
> recovery from security incidents. Explicit security policies,
> security auditing, the design and testing of backup systems, automatic
> and manual log analysis, handling of dialup access, physical security
> for the network, the special problems posed by laptop users, security
> training and documentation, and disaster recovery and costing are
> necessary parts of such an effort.
>
>
>
> > we've got a box on the internet using a speakeasy
> > IP address. a linksys home router sees the front end
> > and NATs traffic for ssh and http to the box, which
> > is a node on the LAN running ubuntu server 8.10.
> > crackers regularly knock on the door. we've
> > implemented IP tables, though they don't work as we
> > think they should. for example:
> >
> > we have a rule (one of many similar)
> > -A INPUT -p tcp -m iprange \
> > --src-range 110.0.0.0-126.255.255.255 \
> > --dport 22 -j DROP
> >
> > iptables -L shows
> > DROP tcp -- anywhere anywhere source IP range \
> > 110.0.0.0-126.255.255.255 tcp dpt:ssh
> >
> > and yet /var/log/auth.log shows ssh login attempts
> > for a variety of user names from
> > 124.93.200.34
>
> You're impliedly classifying this as a problem. I question whether it's
> actually a threat in any meaningful sense, even though: Any box exposed
> to Internet traffic on any port is going to get probed, many, many times
> a day. As part of that, anyone running sshd will get the Internet
> equivalent of twisting the doorknob, which is basically what you've
> described above, many, many times each and every day.
>
> For analogy's sake, let's say you visit a nice city and stay in a hotel.
> You go out, but carelessly leave your room door unlocked. When you
> return, you've been burglarised.
>
> Dismaying, to be sure. However, is it appropriate to go shopping for
> complex door-management hardware? Wouldn't it be smarter, understanding
> as you do the threat model of people twisting hotel doorknobs and
> burglarising rooms that have been carelessly left unlocked, to just not
> leave your door unlocked in the future?
>
> In the case of *ix systems, there are system cracklibs, i.e., the system
> can and should insist that people use strong passwords whenever they run
> /usr/bin/passwd.[1] the /etc/pam.d/common-password PAM configuration
> file controls this. You probably have something like this, there:
>
>
> # /etc/pam.d/common-password - password-related modules common to all
> # services
> #
> # This file is included from other service-specific PAM config files,
> # and should contain a list of modules that define the services to be
> # used to change user passwords. The default is pam_unix
>
> # The "nullok" option allows users to change an empty password, else
> # empty passwords are treated as locked accounts.
> #
> # (Add `md5' after the module name to enable MD5 passwords)
> #
> # The "obscure" option replaces the old `OBSCURE_CHECKS_ENAB' option in
> # login.defs. Also the "min" and "max" options enforce the length of the
> # new password.
>
> password required pam_unix.so nullok obscure min=4 max=8 md5
>
> # Alternate strength checking for password. Note that this
> # requires the libpam-cracklib package to be installed.
> # You will need to comment out the password line above and
> # uncomment the next two in order to use this.
> # (Replaces the `OBSCURE_CHECKS_ENAB', `CRACKLIB_DICTPATH')
> #
> # password required pam_cracklib.so retry=3 minlen=6 difok=3
> # password required pam_unix.so use_authtok nullok md5
>
> As the system says, you probably want to install the libpam-cracklib
> package and uncomment the two lines in the above file, to enable it.
> This page has full details:
> http://www.deer-run.com/~hal/sysadmin/pam_cracklib.html
>
> > the box has been cracked once already, we fixed
> > that vulnerability (i didn't think about a well-known
> > default user, ubuntu: someone guessed that user and
> > password, which was probably a well-known default).
>
> So, you did something very unwise and got caught napping. Don't feel
> too bad; in 1995, I (similarly) made the mistake of assuming the Debian
> AWstats package has security-safe defaults, and got my Web front page
> defaced. At the time, I was so surprised and panicked that, in
> response, I (incorrectly) assumed the server was root-compromised and
> rebuilt it from scratch, unnecessarily.[2] See:
> http://linuxmafia.com/news.html
>
> By the way, your above wording raises a couple of concerns:
>
> 1. Does the term "cracked" in this context entail _root_ compromise?
> I.e., did the intruder merely enter the system masquerading as a
> legitimate user ("ubuntu")[3] whose password he/she guessed -- or, more
> likely, whose automated script guessed that password -- or did the
> intruder do that _and_ then escalate privilege to root-user level or
> equivalent?
>
> That, you see, is a really vital question. If you don't know, you
> really, really need to find out, before doing anything else. If you
> still cannot tell after examination, then the safest course of action is
> to assume root compromise and do a ground-up system rebuild[4], which is
> of course painful. The experience of that level of pain can be, in
> itself, instructive. In 2005, it taught me to be skeptical about the
> security of Web apps, for example. ;-> (Eventually, it also taught me
> a few pointers about how a set of system Web pages might be compromised
> without system _root_ compromise, but at that point I'd already slightly
> overreacted and done a full rebuild. Erring on the side of pessimism in
> such matters is unfortunate but probably wiser than the opposite error.)
>
> 2. If your system _did_ get root-compromised, does your phrase "fixed
> the vulnerability" mean you did a ground-up system rebuild? Because,
> after root compromise, you really have no alternative but to shut the
> system down, secure a copy of the datafiles only, grab an archive set
> of the system configuration files and user dotfiles for reference only
> but not re-use, rebuild software from trusted media, reconstruct system
> configuration files manually referring to the archive set of old
> conffiles, restore users' data but not their dotfile directories or
> executables, set all new passwords, study the old configuration to
> ensure that similar threats are not a problem, post a system bulletin
> explaining what happened, and convey to all users their new passwords
> out-of-band.
>
> If you do anything less than that (following root compromise), you
> haven't fixed the problem.
>
> I stress the above because I've noticed that many newcomers to *ix
> aren't taking that situation seriously. They either don't even try to
> determine, following a security incident, whether the system is now root
> compromised, and just hope for the best, or seriously think that a
> root-compromised system can be "fixed" through some means short of a
> ground-up rebuild.
>
> And, anyway, _who_ created "default user ubuntu"? I really doubt that a
> default Ubuntu install creates any such "joe account", as that would be
> a ghastly design error.
>
> I suspect a system-local human screwup, e.g., someone with admin
> privilege deciding that it'd be a good idea to have login account
> "ubuntu" with password "ubuntu".
>
> Basically, your enforced system policy should preclude creation of "joe
> accounts". Any.
>
>
>
> > ideas we have include
>
> If you'll pardon my airing an opinion based on long years observing such
> discussions, most users' "ideas", and many programmers' ideas, about
> security are bad ones.
>
> The root cause of that badness is failure to start with threat models:
> That term refers to a tracing out of a particular potential threat to a
> system as to mechanism, severity, what's at risk, credibility (is it
> even worth worrying about?), and so on. You cannot rationally start
> taking measures until you understand what you're taking measures
> against, and why, and how.
>
> And, to understand threats against your system, you must first and
> foremost understand your system.
>
> Many users' "ideas" boil down ultimately to little more than technophile
> gadget-freakery, throwing additional software semi-randomly at poorly
> comprehended threats, and thereby making the system more difficult to
> understand and its behaviour more difficult to predict. Many of those
> "ideas" also, themselves, give rise to _new_ threats against the system
> (often but not always denial of service threats).
>
> Avoidable complexity is, in general, bad from several perspectives
> including security. And most users' instinct, when you ask them for
> security suggestions, is to add additional layers of software, which of
> course means greater complexity.
>
>
> > * mount most filesystems in read-only mode (excepting
> > /var/log/, which is a separate mount point)
>
> This is useful, but you should carefully consider why, e.g., what
> purposes does it serve and not serve. You also might, in a similar
> sense, set some mount _options_ for specific parts of the file tree,
> limiting what can be done with/to them.
>
> Here's the partition map for the current, 11-year-old linuxmafia.com
> box[4]:
>
>
> # cat /etc/fstab
> # /etc/fstab: static file system information.
> #
> # <file system> <mount point> <type> <options> <dump> <pass>
> proc /proc proc nosuid 0 0
> /dev/sda5 / ext3 defaults,errors=remount-ro 0 1
> /dev/sda1 /boot ext2 nodev,nosuid,noexec,ro 0 2
> /dev/sdb1 /home ext3 nodev,nosuid 0 2
> /dev/sdb5 /tmp ext2 noatime,nodev,nosuid 0 2
> /dev/sdb8 /usr ext2 nodev,ro 0 2
> /dev/sda9 /usr/local ext3 defaults 0 2
> /dev/sdb7 /var ext3 noatime,nodev,nosuid 0 2
> /dev/sda8 /var/log ext2 noatime,nodev,nosuid 0 2
> /dev/sda7 none swap sw 0 0
> /dev/sdb6 none swap sw 0 0
> /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
> /dev/sda6 /mnt/recovery ext2 rw,noauto 0 2
>
>
> # fdisk -l /dev/sda
>
> Disk /dev/sda: 9105 MB, 9105024000 bytes
> 64 heads, 32 sectors/track, 8683 cylinders
> Units = cylinders of 2048 * 512 = 1048576 bytes
>
> Device Boot Start End Blocks Id System
> /dev/sda1 * 1 95 97264 83 Linux
> /dev/sda2 96 7369 7448576 5 Extended
> /dev/sda5 96 1049 976880 83 Linux
> /dev/sda6 1050 1526 488432 83 Linux
> /dev/sda7 1527 1647 123888 82 Linux swap / Solaris
> /dev/sda8 1648 2601 976880 83 Linux
> /dev/sda9 2602 7369 4882416 83 Linux
>
>
> # fdisk -l /dev/sdb
>
> Disk /dev/sdb: 9105 MB, 9105024000 bytes
> 64 heads, 32 sectors/track, 8683 cylinders
> Units = cylinders of 2048 * 512 = 1048576 bytes
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 954 976880 83 Linux
> /dev/sdb2 955 6129 5299200 5 Extended
> /dev/sdb5 955 1240 292848 83 Linux
> /dev/sdb6 1241 1361 123888 82 Linux swap / Solaris
> /dev/sdb7 1362 3268 1952752 83 Linux
> /dev/sdb8 3269 6129 2929648 83 Linux
>
>
> Now, some of that's a bit antiquated, but you might find some useful
> ideas in there -- including noting what is _not_ done. I found out
> through trial and error, for example, that it's a bad idea to set the
> noexec mount option for /tmp on an ongoing basis, because package
> postinst scripts get run from there. (Which is a pity, since otherwise
> one would not want executables to run, there.) I am tempted to set
> noexec on /tmp, and create an "apt" hook to remount it temporarily to
> allow the exec bit during package operations only, and then remount
> afterwards. (I already have such a hook for the /usr read-only mount.)
>
> What purposes it does serve: The read-only mounts protect against the
> biggest single threat to your system, which is _you_, the sysadmin.
> That is, I (and processes I choose to run) cannot accidentally clobber
> /usr or /boot as the root user, because I must remount those filesystems
> read-write before I can screw with them.
>
> Also, there are performance advantage to some of the options and
> techniques mentioned, e.g., jettisoning the atime stamp on filesystems
> that don't need it, and eschewing the overhead of an ext3 journal on
> filesystems that either are disposable (/var/log, /tmp) or are normally
> read-only (/usr, /boot).
>
> What purposes it does not serve: A non-automated intruder with
> root-level access would not be meaningfully impeded by any of the
> partitioning and mount-option controls cited above, because, having root
> access, he/she can remount anything as desired, trivially.
>
> However, some of the read-only and restricted options cited above
> _could_ very well sabotage many automated attacks if, e.g., in a moment
> of weakness I deployed a vulnerable PHP app. Consider, for example, the
> Lupper worm of November 2005, as I described it in
> http://linuxmafia.com/~rick/faq/index.php?page=virus#virus5 : If
> triggered remotely via one of the criminally vulnerable target Web
> codebases, the remote attack used system calls to run wget, and fetch an
> exploit binary to /tmp, and then run that executable. Setting noexec on
> /tmp (which I currently do not do, but am considering again) would
> therefore prevent the exploit phase of that canned attack.
>
>
>
> > * have sshd listen on some upper port rather than 22
> > (and change iptables rules accordingly)
>
> This is a very popular, classic bad idea: It's one of the canonical
> examples of "security through obscurity", and really gains you nothing
> worth having, when seen in reasonable context.
>
> linuxmafia.com's sshd has been on the Internet since the invention of
> ssh, _on_ port 22 (along with several others, so that particular users
> can have ingress despite stupid corporate firewalls on their end that
> block their connection to outbound port 22), and gets doorknob-twisting
> checks for "joe accounts" from random locations on the Internet many
> times each day. No security compromise has ever resulted, nor would I
> expect any, because I can do the math.
>
> It's simply not possible to hit up my sshd with a rapid enough barrage
> of guessed passwords even for known logins such as "rick", to have any
> reasonable likelihood of success in a reasonable time span (like, years).
> Even if it were, I'd notice my system falling over because /var/log/*
> had overfilled, long before an attacker had even a tiny chance of
> success through such a "dictionary attack" on my sshd.
>
> Thus, with all respect to Prof. Verma and Tom Haddon, I deliberately
> eschew not only running sshd only on non-standard ports but also things
> like DenyHosts and Fail2ban, as pointless because they attempt to solve
> the wrong problem. The correct problem to solve is not "How do I
> prevent continual doorknob-twisting?", but rather "How do I ensure that
> doorknob-twisting doesn't work?"
>
> It doesn't work on linuxmafia.com because the system has no "joe
> accounts", and because dictionary attacks against an sshd statistically
> have no prayer of working unless such accounts are present.
>
> > * have a cron job run every five minutes to monitor
> > the box, mainly checking for weird user activity and
> > probably shutting down the box upon discovering such.
> > * /etc/hosts.deny has ALL=PARANOID and some ip addresses
> > that crackers have used on us.
>
> "Some IP addresses that crackers have used on us"? C'mon, that's just
> dumb. You have no hope of keeping up, and they have an effectively
> unlimited supply of IP addresses, given that many of them are using
> zombified MS-Windows desktop boxes as attack vectors.
>
> You really should consider spending the same amount of effort ensuring
> that there's nothing on the local system that easily attackable, instead
> of trying to wall off potential outside points of origin for attacks
> against local vulnerabilities.
>
>
> > we're not happy with our ideas as a complete defense and hope some
> > of you will chime in with opinions about our ideas as well as ideas
> > we haven't thought of.
>
> OK. Start over.
>
> The ideas you've started with are steps in the wrong direction.
>
> Make sure you understand your system, its weak points, and its probable
> avenues of attack. Make sure you've addressed _all_ aspects of the
> problem: prevention, detection, damage reduction, defence in depth,
> hardening, identification of the attackers, and recovery.
>
> Some stuff from the guy I shave:
>
> http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html
> (You _do_ have a planned-out backup/restore routine, right?)
>
> http://linuxgazette.net/issue98/moen.html
> http://www.itworld.com/print/36437
> http://linuxmafia.com/faq/Security/breakin-without-remote-vulnerability.html
> http://linuxgazette.net/118/weatherwax.html#1
>
>
> [1] Note that this is not effective against users wielding root
> authority including root-equivalent privilege using sudo.
>
> Also note that it doesn't preclude people using the same credentials on
> your system as they do elsewhere, which can lead to user-level breakin
> via stealing of the credentials elsewhere. Again, see
> http://linuxmafia.com/faq/Security/breakin-without-remote-vulnerability.html
> (***COUGH*** shells.sourceforge.net ***COUGH***).
>
> [2] Rebuilding from scratch means that you assume no executable or lib
> or configuration file or dotfile can any longer be trusted, because you
> estimate that a hostile party has gained root-user access and must
> assume he/she has subverted the system. Everything else that's
> necessary to the task of rebuilding follows from that assumption.
>
> [3] There really should not have been such a user. A *ix system should
> not have "role" accounts: You want individual accountability. This
> general rule does not, however, rule out per-system-process usernames
> that the processes run as, such as postgres, bind, sshd, etc. The
> latter should generally have their login shells set as /bin/false in
> /etc/passwd, so that they're not even theoretically possible as shell
> users.
>
> [4] Well, the hardware's 11 years old. I moved the site to that
> hardware around 6-7 years ago.
>
>
> _______________________________________________
> sf-lug mailing list
> sf-lug at linuxmafia.com
> http://linuxmafia.com/mailman/listinfo/sf-lug
>
More information about the sf-lug
mailing list