[sf-lug] HeartBleed - correcting the vulnerability on Linux, etc.
Michael Paoli
Michael.Paoli at cal.berkeley.edu
Sun Apr 13 08:52:57 PDT 2014
From some information I sent out to some select folks earlier last week
(and with some minor bits added/updated, and some bits snipped out).
The bit I'll also additionally note, that I didn't note earlier, for
that particular host, I had also had the luxury of being able to rather
conveniently take it off the network - so once the patches/updates were
downloaded and installed, I took the host off the network, until the
vulnerability had been completely dealt with and taken care of for that
host (and thus even went a bit beyond just covering HeartBleed). I'll
also note that subsequent to this write-up, I did later find that at
least one of the SANS webcasts detailed a relatively similar approach on
Linux (likewise using /proc filesystem, but slightly different approach).
In case you're interested - example remediation of "heartbleed" (OpenSSL
CVE-2014-0160) security bug
Example remediation for "heartbleed" (OpenSSL CVE-2014-0160) security bug
In this particular example case, host is
$ echo $(lsb_release -d; uname -m)
Description: Debian GNU/Linux 7.4 (wheezy) x86_64
Procedure would be fairly similar for other Linux and Unix and similar.
After "patching" - specifically updating the relevant packages per the
distribution security advisories, we're still left with the issue of
running services/PIDs. Until those PIDs go away, the vulnerability
remains (or at least potentially remains) with possibly any and/or all of
those remaining PIDs. One way to do that is reboot. In this particular
case, I decided, as exercise, to see if it was quite feasible to cover
that without rebooting (worst case I could always reboot if that ended
up being necessary or most appropriate or desired way to cover that).
So I check. Most notably, what are the files in the package, and what,
if anything, is still using the files from the older package. As those
have been replaced, anything still using the older files should show up
as using them as unlinked open files, as they've been replaced.
(Note/recall, that for Unix/Linux, if a file is rm(1)ed (unlink(2)ed),
it is removed from the (parent) directory and the link count on the file
is decremented, and if the link count goes to zero, it is deallocated
after no process(es) have the file open. But if the file is still open,
even with link count zero, it continues to exist (it's just not in any
directory at that point) until the last PID having it open closes it (or
the PID goes away, which also closes it)).
Depending upon one's *nix flavor, there are various ways of doing this.
The lsof utility (OpenSource) is a handy way to do this, but it can also
be done by use of the /proc filesystem (yes, even on Solaris - and
probably also AIX - though the procedure is a bit different due to
variations in implementations of the /proc filesystem). HP-UX does some
particularly funky bits with unlinked open files - at least that are
part of a running executable - not going to address that here.
Anyway, in this case I use lsof. I was a bit lazy (/efficient) here -
could've used options to lsof to more narrowly limit to what I was
specifically interested in, but instead I optimized more for my time and
didn't bother with that, and use wee bit of post-processing of lsof
output to filter down to what I was interested in.
So, essentially a (long) one-liner "throw away" script (not important
enough that it ever got saved to a script file executable). And the
command editing histories (e.g. fc, etc.) in bash or similar (e.g. ksh)
shells come in quite handy - quickly build up the desired command,
running through testing some of each of the individual pieces first. I
quickly then have the command shown below, and its output. It
essentially gets listing (from the package management software) of all
files in the impacted package (in this case the pathnames are same in
older and newer replacement, so I can use the listing from the newer
replacement just as well, which is what I do here) - I filter that down
to just the binary files (the only paths omitted were directories and
documentation files) - using grep to filter to where the binaries are
located and find to remove directories and to only display (-print)
ordinary files - thus leaving list of the pathnames of the binary files
of interest (all are library files for the package in this case). We
then make use of awk and the /proc filesystem to look up (displaying
with ls) the binary executables that have those libraries open (and also
use awk and sort -u to filter to listing of PIDs for the processes and
deduplicate our PID listing).
With the listing of the paths of the executing binaries in hand (and
also their PIDs), we can fairly quickly and easily track down what needs
to be restarted.
# (cd /proc && ls -ld $(lsof | fgrep "$(find $(dpkg -L
libssl1.0.0:amd64 | sort | grep '^/usr/lib') -type d -prune -o -type f
-print | sort)" | awk '{if($4=="DEL")print;};' | awk '{print $2;}' |
sort -u | sed -e 's/$/\/exe/'))
ls: cannot access 25832/exe: No such file or directory
lrwxrwxrwx 1 michael users 0 Apr 5 23:32 12711/exe ->
/usr/lib/chromium/chromium
lrwxrwxrwx 1 michael users 0 Apr 5 23:32 23680/exe -> /usr/bin/pidgin
lrwxrwxrwx 1 michael users 0 Apr 5 23:32 25792/exe -> /usr/bin/ssh (deleted)
lrwxrwxrwx 1 michael users 0 Apr 5 23:32 25869/exe -> /usr/bin/ssh (deleted)
lrwxrwxrwx 1 root root 0 Apr 5 23:32 3625/exe -> /sbin/zfs-fuse
lrwxrwxrwx 1 bind bind 0 Apr 5 23:32 4271/exe -> /usr/sbin/named
lrwxrwxrwx 1 root root 0 Apr 5 23:32 4287/exe -> /usr/sbin/lwresd
lrwxrwxrwx 1 root root 0 Apr 5 23:32 4375/exe -> /usr/sbin/ntpd
lrwxrwxrwx 1 root root 0 Apr 5 23:32 4556/exe ->
/usr/lib/sm.bin/sendmail
lrwxrwxrwx 1 root root 0 Apr 5 23:32 4568/exe ->
/usr/bin/python2.7 (deleted)
lrwxrwxrwx 1 root root 0 Apr 5 23:32 4922/exe -> /usr/sbin/libvirtd
lrwxrwxrwx 1 colord colord 0 Apr 5 23:32 4980/exe ->
/usr/lib/x86_64-linux-gnu/colord/colord-sane
lrwxrwxrwx 1 root root 0 Apr 5 23:32 5407/exe ->
/usr/bin/ssh-agent (deleted)
lrwxrwxrwx 1 michael users 0 Apr 5 23:32 6866/exe ->
/usr/lib/xulrunner-24.0/plugin-container (deleted)
#
We can ignore that bit of stderr diagnostic we got from ls - our /proc
is a bit of a moving target, and one PID was gone before ls tried to
give us the binary executable information on it from the /proc
filesystem.
restarted various services (ssh, apache2, bind9, ntp, sendmail, lwresd,
fail2ban, libvirt-guests, libvirt-bin, zfs-fuse, cups)
Got users to restart their processes - not all, but specifically those
relevant and found and covered: ssh, ssh-agent, chromium, iceweasel,
pidgin
Also used a Debian specific recommended tool, and found one more service
to restart:
# checkrestart -p libssl1.0.0:amd64
(1 distinct packages)
Of these, 1 seem to contain init scripts which can be used to restart them:
The following packages seem to have init scripts that could be used
to restart them:
cups:
4911 /usr/sbin/cupsd
These are the init scripts:
service cups restart
#
And restarted that (cups) service (noted above, but didn't find and
restart it on the initial pass).
After that, repeating our check finds nothing left:
# (cd /proc && ls -ld $(lsof | fgrep "$(find $(dpkg -L
libssl1.0.0:amd64 | sort | grep '^/usr/lib') -type d -prune -o -type f
-print | sort)" | awk '{if($4=="DEL")print;};' | awk '{print $2;}' |
sort -u | sed -e 's/$/\/exe/'))
dr-xr-xr-x 336 root root 0 Feb 24 04:38 .
#
That one directory output we get is essentially a false positive (result
of ls -ld with no non-option arguments).
On this host, I nominally have /usr mounted read-only :-) That has many
advantages (e.g. security, slight performance advantage, keeps root (/)
filesystem more reasonably/minimally sized, and don't ever get bloody
alarms or the like with somebody/something filling up /usr filesystem by
writing stuff there (per FHS - see:
http://www.pathname.com/fhs/pub/fhs-2.3.html#THEUSRHIERARCHY
stuff beneath /usr is "is shareable, read-only data", and thus can be
read-only most of the time (e.g. when not performing system maintenance
such as patches/updates, or installation/removal of operating system
software)). Note also, on a dpkg apt based system (e.g. Debian and
most or all derivatives), one can configure apt with post- and pre-
commands, so, e.g., I have this host set up to remount /usr rw before it
attempts to update software with apt-based tools, and after it remounts
/usr ro (or at least attempts to, and the mount command will cause a
diagnostic to be written to stderr if that remount ro attempt fails).
Note also that some OS flavors may not support this. Not only does yum
(at least last I checked) have no capability to put in such pre- and
post- hooks, but current Fedora disallows /usr from being a separate
filesystem from root (/) :-( - I think that's really laziness on the
part of some distributions/vendors - to keep them separate, one needs
determine what must be on root (/) filesystem for single user mode, and
be sure that's all present there, vs. what may be on/under /usr and need
not be present in single user mode. Well, I think some
distributions/vendors quite gave up on that (e.g. Fedora, Solaris, AIX),
and either require /usr be same filesystem as root (/), or require /usr
must also be present/mounted for single user mode. Anyway, since Fedora
is effectively Red Hat Beta, expect Red Hat to generally go in that
directly too - not allowing /usr to be a separate filesystem from root
(/).
Anyway, at this point the bit for "heartbleed" (OpenSSL CVE-2014-0160)
of reboot host or restart all impacted services & PIDs, has been covered
(done via the latter in this case).
While I'm on a roll, I check to see if I can get /usr remounted ro (it
had failed earlier).
# mount -o remount,ro /usr
mount: /usr is busy
#
I check and that fails, so I check further for other things that may be
preventing that from being remounted ro.
# checkrestart
Found 0 processes using old versions of upgraded files
# lsof | fgrep DEL | fgrep /usr
/usr/sbin 4815 root DEL REG 0,4
56496783 /dev/zero
/usr/sbin 4820 www-data DEL REG 0,4
56496783 /dev/zero
/usr/sbin 4821 www-data DEL REG 0,4
56496783 /dev/zero
/usr/sbin 4822 www-data DEL REG 0,4
56496783 /dev/zero
/usr/sbin 4823 www-data DEL REG 0,4
56496783 /dev/zero
/usr/sbin 4824 www-data DEL REG 0,4
56496783 /dev/zero
gnome-ter 26658 michael DEL REG 254,9
115908 /usr/share/icons/hicolor/icon-theme.cache
gdbus 26658 26661 michael DEL REG 254,9
115908 /usr/share/icons/hicolor/icon-theme.cache
dconf 26658 26663 michael DEL REG 254,9
115908 /usr/share/icons/hicolor/icon-theme.cache
gmain 26658 26670 michael DEL REG 254,9
115908 /usr/share/icons/hicolor/icon-theme.cache
#
The /dev/zero bits in the above are false positives, but not the other
PID:
# ps lwwwwwwp 26658
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
0 1003 26658 5472 20 0 356400 9192 - Sl pts/0 96:55
gnome-terminal
#
I get the user to restart their gnome-terminal session (noted above, but
didn't find and have that terminated/restarted on earlier pass), and then:
# mount -o remount,ro /usr
#
The remount ro of /usr succeeds. At that point we're ensured we have no
open unlinked files on /usr (and no files open in a mode that allows
writing or appending) - thus we've fully covered the PID issue for the
/usr filesystem - which happened to contain all the binaries for the
"heartbleed" (OpenSSL CVE-2014-0160) security bug issue on this host.
So we're mostly done. The only bit that remains is remediation of any
potentially exposed data (any contents that may earlier have been read
from memory). We'll address that other ways (beyond the scope of what
I'm writing up here).
More information about the sf-lug
mailing list