[sf-lug] Hacked RHEL4/PHP4 server

[Tom Haddon]

Hi Folks,

I'm hoping I can marshall the resources of the LUG to help me get a
hacked server back under control. Here's the situation...

I used to work for a non-profit back in the early 2000's that did health
resource information, and while working for them I wrote my first ever
web application using PHP/PostgreSQL. It was a cancer resource guide,
showing what resources were available to patients and families in a
local area.

Needless to say it was terribly coded, but it worked. And worked fine
while I was at the non-profit, and for a good few years after I left
until the non-profit went out of business. Then I was approached by
another non-profit that had been using the web application (it was
skinnable, so other orgs could have their own version) and asked I could
help them move the site to their own server as they'd like to continue
using it. So I did, about a year or so ago. They have a Virtual Private
Server with Verio that runs RHEL4 and I have an account with root access
on this. I'm not really the administrator for this server (although I do
have root access) - I was paid just to setup the website and make sure
the backups and such were in place. I was told that Verio would be
responsible for the day to day management of the server, but I'm not
sure how true that is. Other vendors do have access to the server, and
make changes to other websites hosted there. 

Today I got an email from them that Google had removed them from their
search listing because there was a whole bunch of pharmacy spam embedded
in their website. This was in the form:

<div id="localnews" style="position:absolute; left:0px; top:0px;
height:1px; overflow:hidden;">
<a
href="http://EXTERNAL_URL_OF_WEBSITE/5-cheapest-phentermine-prescription.html" title="cheapest phentermine prescription">cheapest phentermine prescription</a>
[..]
[..]
</div>

This was embedded in a header page that is included in all pages listed
on the website. It's a custom written application, so they would have
had to get fairly familiar with the layout of the site to know which one
to modify to have the most affect. As I'm writing this I'm suspecting
that this suggests they had shell access to the server to be able to
inspect files on the filesystem, whereas when I was investigating it, I
assumed they exploited some PHP bug and got in that way. 

What I've done so far:

- Removed the offending content from the header page (the bit I've
pasted above)
- Changed permissions on all the files and folders in the DocumentRoot
to be owned by root and 644 perms.

I think the change was made around the 5th May, from looking at the
apache logs. Looking at the "last" command, there was no access to the
server around then. There was ftp user access around the 25th April, but
after that the next access is me.

So there are a few things confusing me:

- The link that was each of these items was linking to was on the same
host as the site itself, but I still can't find the pages that it's
linking to on the filesystem. I've confirmed they are getting 200 OK
responses in the apache log, so I know it's not a 404 error that's being
redirected somewhere, or that it's actually being served from another
server. But I can't find the files themselves. So in the example above,
I go to the DocumentRoot and look
for 5-cheapest-phentermine-prescription.html but can't find anything. I
grep for some content that I know is on that page, but nothing. Any
ideas on how I can debug where/how these files are being served from?
- How did they actually modify the content? Through a PHP exploit, or
shell access?

Any help appreciated.

Thanks, Tom