[sf-lug] traffic shaping, ...: Re: [on-list] site up, http[s] down: Re: Wierd problems trying to access linuxmafia.com

[Rick Moen]

Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):

> There are traffic shaping packages for at least most major Linux
> distributions.

Yes, I'll be looking at those.  As usual, there is an enlightening Gentoo
wiki page about a specific implementation of this idea that uses only
kernel code and iproute2's 'tc' (traffic control) utility:
https://wiki.gentoo.org/wiki/Traffic_shaping

..and a similar Arch Linux wiki page:
https://wiki.archlinux.org/index.php/advanced_traffic_control

The Gentoo page combines the kernel Fair Queue Controlled Delay AQM
(FQ_CODEL) module to create fair bandwidth for all flows, while
attempting to minimise buffers (and hence delays) with throttling of
total upload and download speeds to about 90% (adjustable, obviously) of
one's maximum link speed using several kernel QoS modules and the
iproute2 'tc' utility.

I'm thinking that's probably what I _most_ need, because of the rather
unfortunate characteristics of aDSL:  The outbound capacity is small
compared to inbound, with the result that requests can saturate outbound
and then effectively everything gets clogged for new traffic such as,
oh, say, my ssh connections to my server, at which point I currently get
peevish and start shooting services in the foot.  ;->


But let's also consider the uplink.

My Raw Bandwidth Communications (RBC) invoice states the _nominal_ 
bandwidth I pay for:

'Residential DSL 384K-1.5M/128-384K monthly w/circuit'.

I take that to mean 384 kbps to 1.5 Mbps _downwards_ bandwidth, 128 kbps to
384 kbps _upwards_ bandwidth.  Of course, the actual bandwidth inevitably
differs and in my case is probably lower on account of high distance
from the telco central office (AT&T LATA1, central office aka Serving
Wire Center MNPKCA11 [1]) at 2950 Sand Hill Road (nr. I-280), Menlo Park,
about 2.6 to 4.0 km (1.6 to 2.5 miles) away, depending on which streets
the copper cabling follows.  As per the chart on
https://www.increasebroadbandspeed.co.uk/2012/graph-ADSL-speed-versus-distance
, ADSL speed starts dropping at 2.5 km (1.5 miles) and peters out about
5 km (3.1 miles).

What I probably should do in the near future to get (at least at one
sampling time) the _real_ effective total bandwidth up and down is shut
off all network services on my server, then use something like
http://www.speedtest.net/ to measure real throughput up and down -- as a
baseline for subsequent throttling.

No time like the present, actually.  So:

Speedtest.net via RBC aDSL to Speedtest's server at Monkey Brains in San Francisco:

0.65 Mbps (665 kbps) down
0.07 Mbps (72 kbps) up
62 ms ping times

For purposes of _that_ test, I made linuxmafia.com be quiescent by
temporarily shutting down apache2, exim, rsyncd, vsftpd, ntpd, and
bind9, leaving only sshd listening.  However, on reflection I worried: 
Suppose there were hidden services serving bad guys, or other host weirdness?  
So:

Repeating that test with my server completely severed from the RBC aDSL, 
i.e., yanking its ethernet cable (to baseline this measure and
double-check for possible server mischief):

0.66 Mbps (675 kbps) down
0.06 Mbps (62 kbps) up
71 ms ping times

That's so close as to be well within margin of error, further supporting
my continuing working hypothesis that the linuxmafia.com server has
_not_ been security-compromised.  (Yay.)



However, while I was at it, it occurred to me to do additional test runs
that eliminated not only my linuxmafia.com server as a bandwidth hog but
also my entire inside LAN (an Arduino watering system, an Apple Airport
Extreme WAP, some aging ethernet cabling, and an ethernet switch), by
connecting my laptop (with static IP) on ethernet directly to the aDSL
bridge box, and disconnecting my entire house network -- leaving as
local suspects only the house landline (AT&T's problem along with
everything else outwards from the demarc), the demarc box, the
RBC-supplied high-pass filter widget that splits aDSL digital from
landline voice, the RJ-11 cable from there to the aDSL bridge box, the
(Westell) aDSL bridge box, and the short and apparently undamaged
ethernet cable to my laptop.  (The only one of those components that I
suspect could in theory be dodgy is the RJ-11 cable.)

I was able to get one additional good test run to Speedtest's server at
Monkey Brains, but then I gather that I must have triggered capping on my
IP as I was getting partial and then complete test failure, and had to
switch around to other S.F.-based Speedtest servers -- but, with outlier
data points in two cases for upbound, the results were consistent:

Still using Speedtest's server at Monkey Brains in San Francisco:
0.61 Mbps (624 kbps) down
0.08 Mbps (82 kbps) up
60 ms ping times

Same but to Speedtest's Race Communications server in San Francisco:
0.63 Mbps (645 kbps) down
3.96 Mbps (4055 kbps)  up
72 ms ping times

Same but to Speedtest's AT&T server in San Francisco:
0.70 Mbps (717 kbps) down
0.28 Mbps (286 kbps) up
95 ms ping times

Same but to Speedtest's Fastmetrics server in San Francisco:
0.65 Mbps (665 kbps) down
0.07 Mbps (72 kbps) up
68 ms ping times


So, I'm thinking that the last test's set of measures are probably
typical, at least at this hour.  I'm tempted to do more tests to see
if there are substantial variations over time, but have curtailed those
tests for now because it was cold out there.

The Lineus Bottomus:  Correct me if I'm wrong, but everything I'm seeing
so far suggests saturation of the (~72 kbps) uplink.

My server-centric household use-case is the opposite of what aDSL is
good for:  Normal non-server aDSL usage does mostly download traffic
with light traffic back through the uplink.  A server does mostly
upbound traffic with light downwards (e.g., Web request) traffic.  So,
the moment the uplink saturates, everything goes miserable.

https://wiki.sonic.net/wiki/Bandwidth_Saturation
(Page mostly assumes the user-typical balance of traffic, but is
otherwise useful.)  Also:
https://newsignature.com/articles/tcp-adsl-uplink-oversubscription-bad/

   [...]
   To wrap up, it is generally a very good idea, and should be considered
   best practice, to use only synchronous connectivity for business
   customers who have anything more than basic internet connectivity needs.

Yeah, that was why I was so bummed when my prior provider, NorthPoint
Communications, went belly-up in 2001, because it provided residential 
_SDSL_ (Symmetric Digital Subscriber Line), which didn't have this
problem at all.

In conclusion:  QoS / traffic-shaping needed.  And I'll ask Mike Durkin
to mail me a known-good RJ-11 patchcord.



Just for giggles, here's _good_ broadband connectivity as measured to
the same Speedtest server over my mother-in-law's Comcast Business cable
Internet connection:

29.7 Mbps down
12.0 Mbps up
17 ms ping times




> Note also that traffic shaping can be deployed as a pass-through for,
> e.g. a home network, giving benefit - and relative fairness to all -
> though traffic shaping can also be rather to quite useful even if
> deployed on as little as one individual host.

This is one of the advantage of networks having a capable bastion host, 
through which everything flows on the way in and out.  My RBC
aDSL-connected house network lacks such a bastion host, but OTOH only
trivial amounts of traffic in and out normall got to or from anywhere
_but_ my server, so that's really where I'll want to apply some brakes
on impolite bandwidth-sucking.

> As for Apache, I do rather also like the idea of a robots.txt with some
> essentially "honeypot" do not go here stuff (and especially that's not
> otherwise linked to or at all or found by a regular web crawl), and
> use that (via logs, or CGI, or whatever) to have such offending client
> IPs automagically throttled (at least until unused for some while) down
> to the bandwidth of sucking chilled molasses through a cocktail straw.

I like that.  It's sneaky.

You might be as surprised as I was to hear that .htaccess is deprecated
as a place to do such things:  The hit to Apache httpd performance on
each page load is quite considerable, and you're better off putting the
same logic, if at all, into Apache's conffiles, as their contents are
cached at startup.

But I am increasingly thinking the problem at my address has nothing
fundamentally to do with abusive Web traffic.  It's just uplink
saturation.



[1] For a small amusement, try to Internet-research the street address of
_your_ telco central office (for either landline or ADSL/ADSL2/ADSL2+
purposes).  Ever since the year 2001, utilities have rather
pathologically implemented a 'Telling the public anything helps
terrorists' excuse for hiding information about their infrastructure.  
Nonetheless, if you are persistent and peruse regulatory filings at CPUC
and FCC, you can find yours, as I did mine.