[sf-lug] Hacked RHEL4/PHP4 server

[Rick Moen]

Quoting Tom Haddon (tom at greenleaftech.net):

[A non-profit you used to work for ran a custom PHP/PostgreSQL app.
A second non-profit picked up that app, and has been running it on a
Verio virthost.]

> I was told that Verio would be responsible for the day to day management 
> of the server, but I'm not sure how true that is.

Verio, eh?  It... um... could happen.  ;->  (I'd not rely on that.)

[Spam pages are now being served from the virthost, via modified PHP
header files.]

> As I'm writing this I'm suspecting that this suggests they had shell
> access to the server to be able to inspect files on the filesystem,
> whereas when I was investigating it, I assumed they exploited some PHP
> bug and got in that way. 

Seems a reasonable assumption -- although that might very well be just 
remote ability to run shell scripts on the system.  One important
question thus is:  What UID was/is usable by the attacker for that
purpose.  One very bad sign:

> The link that was each of these items was linking to was on the same
> host as the site itself, but I still can't find the pages that it's
> linking to on the filesystem.

So, the most natural inference is that your inability to find it is
because the utilities you're attempting to use for that purpose have
been replaced by trojaned substitutes that have been compiled to have a
built-in inability to see certain processes, users, and files -- e.g.,
ls, netstat, ps, w, who, find, login, du, pstree, killall, top,
syslogd....

What I'm saying is that, if the file is truly being served from the
system's Web server, and yet you as the root user can't find it on the
system, then one reasonable hypothesis would be that the system had its
root account cracked, and then (to hide the intrusion) was rootkitted --
i.e., the above sorts of standard system-monitoring utilities were
replaced by trojaned versions.

You _could_ try copying over, or compiling locally, some known-good
copies of those utilities into /usr/local/bin.  Or maybe, just force
reinstallation of the distro packages that furnish them.  Then (quickly)
check again for those files, etc.

If you have good reason to suspect that the system's been root
compromised, then it should be taken offline summarily, fully backed up,
studied to learn more about the intrusion, and then rebuilt.

The rebuild should (given root compromise) avoid re-using any existing
program code, any existing server configuration files, and any existing
user dotfiles and dotfile directories -- because all of those need to be
considered suspect.  The only files you can just roll forward are
non-program datafiles -- after, of course, getting rid of the Web site
defacement.

What you'd _really_ like to know, other than "Was the system really
root-compromised?" (which is a very important question) is:  "How
exactly did the bad guys get in, and how did they escalate to root-user
access?"  You can certainly look at the logfiles -- if those weren't 
erased -- and look around for other clues.  Most of these guys aren't 
particularly subtle or careful.

You'll want to consider exactly how unmaintained _was_ this virthost that
was [**cough**] managed by Verio [**cough**].  How badly exposed to
unfixed security bugs _were_ the running network daemons?  

> Looking at the "last" command....

That would be yet another utility that would be a natural inclusion for
rootkitting.  So, you could be asking the burglar's own tools how he got
in, and they're saying "Burglar?  I didn't see any burglar."  And a
rootkitted utility sees no evil, hears no evil, speaks no evil, y'see.

If you conclude that it's likely the unit has been root-compromised,
here's your next stop:
http://www.cert.org/tech_tips/win-UNIX-system_compromise.html

-- 
Cheers,                      Evolution:  Life's a niche, and then you die.
Rick Moen
rick at linuxmafia.com