[conspire] Apache2 bandwidth limiting: fixing 1105 Altschul's connectivity

[Sean]

just a scant $0.02:

http://www.cyberciti.biz/tips/howto-prevent-image-hotlinking-leeching-direct-linking.html

I found it very painless and rewarding to switch from Apache to
Lighttpd, YMMV, and this page includes a link to do the same thing with
Apache, though I'm sort of promoting Lighttpd here, FWIW.

OTOH, I once briefly had the inclination to crawl linuxmafia's knowledge
base myself just to have a local copy (an idea I quickliy got over), so
perhaps a prefab tarball to actually allow such a download might eschew
such crawlers if that be their intent.  It is a neat little KB, after all.

OTMA (One Too Many Acronyms),
_S

On 01/11/2012 12:08 AM, Rick Moen wrote:
> CABAL attendees will be happy to hear that we've chased down and fixed
> two major causes for connectivity problems Chez Moen.
>
>
> 1.  Obsolete DNS nameserver IP.
>
> The WAPs we have around here, and also /etc/resolv.conf on my server, 
> have tended to include two IP addresses of DNS nameservers that Raw
> Bandwidth Communications makes accessible to its customers (such as us) 
> for full recursive service:
>
> 198.144.192.2
> 198.144.192.4
>
> That's in addition to my server, IP 198.144.195.186, which is likewise a
> full recursive nameserver.  The problem is, it appears that
> 198.144.192.2 has been retired.  Depending on round-robin
> implementation, maybe something like half of all DNS queries from
> machines with wireless DHCP leases were going to a nonexistent
> nameserver.  The other half were going to my nameserver, which 
> was configured to think that 198.144.192.2 was a valid forwarding IP,
> so some of _its_ traffic was likewise going to nowhere. 
>
> This has all been fixed.
>
>
>
>
> 2.  My instance of the Apache2 HTTPd was wide-open to abuse by bandwidth
> hogs.  The past two days, almost 100% of incoming bandwidth was in use,
> and logfile analysis revealed that substantively everything was being
> grabbed by many thousands of rapidfire requests to Apache from a single
> IP address in Scotland.  Someone there had fired up a mirroring script 
> to spider through my entire site and grab every available file without
> exception:  photos of my vegetable garden, tarballs of obsolete SSH ports,
> pipermail archives, everything without exception.  
>
> Our immediate tactical measure was to blacklist that IP in
> /etc/apache2/sites-available/default:
>
>         <Directory />
>                 Options Indexes FollowSymLinks 
>                 AllowOverride None
>                 Deny from 139.133.7.237 
>         </Directory>
>
> However, if one guy today in Scotland with a terabyte array finds it
> easier to wget my entire site than to target just what he wants, that
> pretty much guarantees that there are a million other idiots just like
> him in various other parts of the world.  
>
> So, I looked at throttling options.  In a follow-up post, I might write
> about other ways of doing this (including the 'tc' / Traffic Control
> software for doing system-wide throttling at the kernel level), but my 
> immediate solution is mod_bw, which is an update/successor to the old
> Apache 1.3 mod_bandwidth module, ported to Apache2.
>
> Two pretty good articles:
> http://www.uno-code.com/?q=node/64
> http://wiki.excito.org/wiki/index.php/Throttle_Apache_bandwidth
>
> The first of those two also covers the _other_ currently popular
> solution, an Apache2 module called mod_cband ('Apache2 bandwidth quota
> and throttling module').
>
>
> Steps taken here:
>
> apt-get install libapache2-mod-bw  #Fetch Debian package.
> a2enmod bw  #Enable the module within the installed Apache instance.
>
>
> Add these lines to the port-80 and port443 VirtualHosts stanzas in
> /etc/apache2/sites/available/default:
>
>
>
> BandwidthModule On
> ForceBandwidthModule On
> Bandwidth all 250000
> LargeFileLimit .mp3 1 30000
> LargeFileLimit .gz  5 30000
> LargeFileLimit .gif 1 30000
> LargeFileLimit .png 1 30000
> LargeFileLimit .zip 1 30000
> LargeFileLimit .pdf 1 30000
> LargeFileLimit .exe 1 30000
> LargeFileLimit .mov 1 30000
> LargeFileLimit .jpeg 1 30000
>
>
>
> Do:
> /etc/init.d/apache2 restart
>
>
> To explain the lines added to the Apache conffiles:
>
> The 'Bandwidth all 250000' line limits total Apache2 bandwidth to
> 250kB/s.
>
> The various LargeFileLimit [foo] 1 30000' throttles down to 30kB/s any
> request for a file with the matching filenamed extension that is over 1
> kB.
>
>
> Lots more things can be done, but I'm starting with the low-hanging
> fruit.   Module documentation:
> http://bwmod.sourceforge.net/files/mod_bw-0.7.txt
>
>
> _______________________________________________
> conspire mailing list
> conspire at linuxmafia.com
> http://linuxmafia.com/mailman/listinfo/conspire
>