[conspire] Apache2 bandwidth limiting: fixing 1105 Altschul's connectivity

[Rick Moen]

CABAL attendees will be happy to hear that we've chased down and fixed
two major causes for connectivity problems Chez Moen.


1.  Obsolete DNS nameserver IP.

The WAPs we have around here, and also /etc/resolv.conf on my server, 
have tended to include two IP addresses of DNS nameservers that Raw
Bandwidth Communications makes accessible to its customers (such as us) 
for full recursive service:

198.144.192.2
198.144.192.4

That's in addition to my server, IP 198.144.195.186, which is likewise a
full recursive nameserver.  The problem is, it appears that
198.144.192.2 has been retired.  Depending on round-robin
implementation, maybe something like half of all DNS queries from
machines with wireless DHCP leases were going to a nonexistent
nameserver.  The other half were going to my nameserver, which 
was configured to think that 198.144.192.2 was a valid forwarding IP,
so some of _its_ traffic was likewise going to nowhere. 

This has all been fixed.




2.  My instance of the Apache2 HTTPd was wide-open to abuse by bandwidth
hogs.  The past two days, almost 100% of incoming bandwidth was in use,
and logfile analysis revealed that substantively everything was being
grabbed by many thousands of rapidfire requests to Apache from a single
IP address in Scotland.  Someone there had fired up a mirroring script 
to spider through my entire site and grab every available file without
exception:  photos of my vegetable garden, tarballs of obsolete SSH ports,
pipermail archives, everything without exception.  

Our immediate tactical measure was to blacklist that IP in
/etc/apache2/sites-available/default:

        <Directory />
                Options Indexes FollowSymLinks 
                AllowOverride None
                Deny from 139.133.7.237 
        </Directory>

However, if one guy today in Scotland with a terabyte array finds it
easier to wget my entire site than to target just what he wants, that
pretty much guarantees that there are a million other idiots just like
him in various other parts of the world.  

So, I looked at throttling options.  In a follow-up post, I might write
about other ways of doing this (including the 'tc' / Traffic Control
software for doing system-wide throttling at the kernel level), but my 
immediate solution is mod_bw, which is an update/successor to the old
Apache 1.3 mod_bandwidth module, ported to Apache2.

Two pretty good articles:
http://www.uno-code.com/?q=node/64
http://wiki.excito.org/wiki/index.php/Throttle_Apache_bandwidth

The first of those two also covers the _other_ currently popular
solution, an Apache2 module called mod_cband ('Apache2 bandwidth quota
and throttling module').


Steps taken here:

apt-get install libapache2-mod-bw  #Fetch Debian package.
a2enmod bw  #Enable the module within the installed Apache instance.


Add these lines to the port-80 and port443 VirtualHosts stanzas in
/etc/apache2/sites/available/default:



BandwidthModule On
ForceBandwidthModule On
Bandwidth all 250000
LargeFileLimit .mp3 1 30000
LargeFileLimit .gz  5 30000
LargeFileLimit .gif 1 30000
LargeFileLimit .png 1 30000
LargeFileLimit .zip 1 30000
LargeFileLimit .pdf 1 30000
LargeFileLimit .exe 1 30000
LargeFileLimit .mov 1 30000
LargeFileLimit .jpeg 1 30000



Do:
/etc/init.d/apache2 restart


To explain the lines added to the Apache conffiles:

The 'Bandwidth all 250000' line limits total Apache2 bandwidth to
250kB/s.

The various LargeFileLimit [foo] 1 30000' throttles down to 30kB/s any
request for a file with the matching filenamed extension that is over 1
kB.


Lots more things can be done, but I'm starting with the low-hanging
fruit.   Module documentation:
http://bwmod.sourceforge.net/files/mod_bw-0.7.txt