[sf-lug] Hacked RHEL4/PHP4 server

[Tom Haddon]

On Thu, 2008-05-22 at 09:12 -0700, Asheesh Laroia wrote:
> On Thu, 22 May 2008, Tom Haddon wrote:
> 
> > On Thu, 2008-05-22 at 07:56 -0700, Kristian Erik Hermansen wrote:
> >> Do all of the injected html links start with a common prefix for the
> >> file?  For instance "5-", like your example ?
> >
> > They all match the pattern [0-9]-(.*)phentermine(.*).html. I'm beginning
> > to think the server must have been compromised after all. I am able to
> > create file of the same name as one of these with a test page and see
> > that it's served in place of the spam page. If I delete the page with
> > rm, the spam page shows up again. If I try to delete it again, it says
> > no such file.
> 
> Are you sure there's no mod_rewrite action going on?  e.g. 
> http://articles.techrepublic.com.com/5100-10878_11-5068743.html discusses 
> RewriteLogLevel - try doing that and going to one of the evil URLs (after 
> disabling the redirect as you discussed).

I don't *think* so. The requests for those pages were getting 200 return
codes in the apache logs, and I've checked the apache config file and
from what I can tell there was no changes to add any rewrite rules.

> 
> Also, what if you just "grep -ri <string_in_one_of_those_html_files> /" ? 
> What about "zgrep -ri <string_in_one_of_those_evil_html_files> /"?  To 
> find the string, load the page up in a browser and look for something 
> fairly unique.

Neither produces anything, but if it has been compromised, I think this
would make sense, as it's hiding the files altogether, so grep wouldn't
see them either. 

> 
> Can you make a copy of the disk image on a different server?  And have you 
> tried asking RPM (which could, I know, have had its database pwned also) 
> to verify the stuff on the machine: 
> http://www.redhat.com/archives/rpm-list/2002-April/msg00118.html ?

Hmm, wasn't aware of that. Will give it a try, thx.

> 
> Doing these would take a lot of time, but they'd be background jobs, so 
> it's only compute time.  Obviously I don't think you should spend your 
> whole life cleaning up after script kiddies when it's not your job to.

Yeah, on the one hand I do want to find out what happened. On the other,
I'm not (yet) being paid to do so. I have a call with the non-profit
early next week to discuss next steps, so that may change, in which case
I'll be doing a more thorough investigation.

> 
> > I've put in place an apache redirect for the matching file types so that 
> > if anyone is going to those URLs, they'll be redirected to the main 
> > site.
> 
> Yay.
> 
> > At this point, I'd have to advise the non-profit per Rick's comments to
> > pretty much start from scratch with this server and/or have verio clean
> > up the mess...
> 
> Sounds reasonable.
> 
> Out of curiosity, which virtualization technology?

Not sure, I'm afraid. This seems a bit vague and marketing-type-speak:

http://www.verio.com/vps-technology/

Cheers, Tom

> 
> Perhaps further discussion on the list could lead us to find a setup more 
> resilient against their attackers but not too onerous for them.
> 
> -- Asheesh.
>