[conspire] How to kill a gigabit switch

[Rick Moen]

Going back on-list.  Hope you don't mind.

Quoting Paul Zander (paulz at ieee.org):

> I would first wonder how the Netgear box was motivated to do a belly
> flop.  Fan vibration from other electronic stuff?

No.  It's more prosaic than that, and it also involved my knowing
neglect of that situation.

I don't know if you remember where this wooden hatch in the wall is.
As you go down the stairs into the garage, the near wall on your left is
the one with the washer & dryer.  Just past the dryer is an alcove next
to the dryer, and there's a removable wooden hatch sideways in the wall,
that extends upwards to about 3' off the concrete garage floor.  This is 
the access into the crawl space under the house's floor.  There's dirt
below, house above.

Years ago, when we moved into 1105 Altschul Ave., and did installfests /
CABAL meetings in the garage, we would use the same AC outlet that
drives the washing machine and garage lights, plus my server, which is
IIRC a 15 amp or 10 amp circuit.  At that time, I had no dryer, so the
separate 220VAC circuit for an electric dryer went unused.  

Anyway, with CABAL people doing stupid things on the AC chain, it was
somewhat common, and very irritating, for some numbnuts to overstress
the circuit and bring down everything including my server.  Which made
me very cross.

So, a couple of wonderful guys from SF-LUG, Alex Kleider and another
gentleman, offered to come down and fix this situation for me.  Generous
people.  True friends and benefactors.  One of them welded some about 8"
tall sets of shelves to fasten to the bottoms of the underflooring, to
hold my server gear, the outside ethernet hub or switch, and my ADSL
bridge (what people call an 'ADSL modem').  At that time, Deirdre also
had a server at home, so it went there, too.

The other guy had real skills as an electrician, so he put together
something ingenious that leveraged the unused 220VAC power
socket.  He bought a matching big-ass plug for that socket, and made a
heavy cable that took 110VAC from the socket (I understood it at the time),
and ran that to an elecrical circuit breaker panel that he mounted
vertically hanging from (again) the subflooring.  And last, the guys
put black cloth on top of the dirt for about 15' in each direction.  The
result was a miniature server room, terribly cramped for ceiling space, 
with an independent power feed.  Also, during the hot summer months,
unlike the garage proper, it gets enough airflow through the underhouse
space that my server won't be cooked to death.

Because none of this work touched the house wiring and existing panel, 
no permit or inspection required.

I was immensely grateful, but had a personal problem plus a technical
one.  The personal one is that although I'm not phobic about either
cramped spaces or spiders, I'm still not happy to be in a crawl space
with an entire house inches over my head, and I have good reason to
think that black widows and possibly brown recluse spiders live under
there.  When I'm forced to go under there by some need, I make a point
of looking very carefully before putting my hands or knees down, and
also shining a flashlight around everywhere to both motivate spiders 
to get clear and to spot any sticking around.

The technical problem I had was a nagging suspicion that there might be
something wrong with the power.  First, Deirdre's server motherboard,
PSU, and hard drive died.  (It was a Shuttle PC lunchpail box sized
gamer box.)  Then, one of my VA Linux Systems motherboards died.

About this time, my ancient washing machine died, and I bought the
current washer & dryer pair.  The dryer uses natural gas.

I don't have much faith in coincidence.  I had to entertain the
possibility that surges or spikes were somehow a problem in the new 
power feed.  So, I temporarily moved my server back out onto the
weak regular AC feed inside the garage, which also was handy because 
of recurring motherboard problem and need to get to the server
physically.

Then, two things new entered the picture.  One, I decided to adopt
belt-and-suspenders measures, and got an outboard AC voltage regulator
unit, which is now between my server and whatever AC it's on.  To be
honest, I actually bought two voltage regulator units.  One is an APCC
unit I bought at Fry's.  Later, I somehow pessimistically convinced
myself without bothering to double-check the box the APCC unit is in,
and believed that I'd bought a UPS rather than a voltage regulator by
mistake.  So, I went to the De Anza College ham-club-staffed electronics
swap very early one Saturday, and bought the voltage regulator I'm now
using, for cheap used.  Much later, I double-checked the APCC still in
its box, and saw that it _is_ a voltage regulator.  So, I have a spare,
or something.

The second thing that happened is lights in my house started getting
much brighter or much dimmer one day, and elecric things in my house in
general started acting weird.  I called in an electrician immediately.
He found that the main power feed at the roof, the line from the pole,
had a broken neutral line.  So, indeed, voltage was wildly going up and
down.  We got PG&E out, stat, and they ran an all-new main power line to
the power demarc point on the roof.

By this point, the digital clock / timer on my oven had gotten wonky
(has never worked right since), and I think this hastened the demise of
my now last-but-one microwave oven.  Otherwise, I think we came out OK.  
_And_ I thought:  This might have been the root cause of the motherboard
failures, rather than it being a problem with the new under-house
wiring, but I can't be sure, so a voltage regulator is very cheap
insurance.

Since then, on a couple of times, especially during wind storms, the
voltage regulator under my server has triggered because of a reported
overvoltage condition.  The regulator goes into a 'take down and stay
down' state until you manually reset it, which is a bit annoying for a 
server, but better than frying hardware.  So, yes, we had a wonky main
AC feed, now fixed, but also we had unavoidable, occasional severe power
spikes resulting from PG&E shifting power during storms & such, and 
either or both of those could have done the hardware killing.

And, of course, sometimes hardware just blows up, too.

Anyway, about the ethernet switch being on the black cloth about 1.5' 
in from the hatch:  This little switch wasn't fastened to anything
(though it could have been).  If tugged by strain on ethernet cables, it
would get yanked off the hanging shelf.  I would see this happen, and
think, well, I could crawl into that space I don't really like being in,
and put it back up.  OTOH, sitting where it is, I can see all of its 
link-status lights really clearly, and can reach it from the outside if
necessary to power-cycle it or plug/unplug a cable.  So, where it was
sitting on the black cloth wasn't ideal, but had functional advantages.


> This being the wettest winter in 20 years, perhaps the water under the
> house has occurred before, but not during the drought years.

This is my suspicion.  Also, it could occasionally have pooled 
water even during the (preceding) drought years, and I wouldn't have
noticed if it went away before I thought to look into the hatch again.

The corner of the foundation in question is, in effect, the furthest
downhill corner of the main house (i.e., excluding the garage).  
I really cannot tell for certain which direction the water came from,
but can say all of the surrounding black cloth 15' or so in each
direction is dry, so it's not moving along the ground surface towards
that corner.

The dirt in that corner is a bit lower than the dirt under the rest of
the house, peering back and to the side.  Here's what I think is
happening:  After a series of back-to-back storms, when the ground 
has started to get really soaked, water is moving through the ground
across my lot as groundwater.  The house is below a saddle in the hills,
and you can see where the water gathers and flows through my lot,
thankfully mostly through the back behind Cheryl's studio outbuilding
and underneath my greenhouse.  I don't think the pooled water seeped in
from a drainspout.  I think it is part of the general flow of
groundwater that's seeping through the ground towards the street from
the hill behind my lot.  I suspect it hits the foundation wall and is
blocked a bit, so gathers and seeps up into a pool at the low spot near
the hatch entrance.

A day and a half ago, I shined a flashlight in and saw that maybe 3-5
gallons of water were pooled there again.  I repeated that check a few
minutes ago, and it now looks dry.  So, I suspect this is just one of
those places that get water pooling whenever the ground is really,
really saturated with the effects of back-to-back Pacific storms.

[snip your point #1 about how rainspouts really ought to put water 10'
from the house.]

That having been said, I'll add to my overflowing agenda seeing about
making sure the rainspouts deposit the water somewhere more distant from
the house's edge.  We'd really like to collect that for the garden,
anyway, but it's been unclear how best to do that (especially without a
mosquito-breeding problem).

> 2. There is a definite slope in the backyard toward the house.   Water
> that lands in the next street is just finding its way down the slope
> and finds your house.  This is a lot harder to correct than #1 and is
> probably totally impractical.  Is there a practical way to drain the
> water under the house toward the street?

Don't think so.  I think we just have to live with it.  Suspect it's
been doing this since 1956, so fine, it's just what happens.

> Some years back, a ham friend lived not far from De Anza college.  One
> January he started digging a hole to put up an antenna tower. About a
> foot down, he hit water.  The hole didn't drain until May.  Basically
> there is an aquifer that starts high in the hills and spreads out
> across Cupertino.  In a wet winter the water table is just below the
> surface.  

Oh yeah.  

First, are you aware of that staggeringly tall coast redwood tree on my
lot?  Mom planted that when I was born, and was blown away by how well
it's done.  That suggests it has reliable water.  It's very close to the
main water flow path down from the saddle behind my lot.

During very heavy rain, you can see the main flow path very clearly
because it starts turning into a stream.  I figured this out shortly
after I moved back in, in 2006 (after being away literally 40 years), 
and experimentally dug ditches at right angles to that path of flow.
In the agriculural philosophy called 'permaculture', these artificial 
topological features are called 'swales', reviving a very old English
word.  The three demo swales have been a roaring success, in that 
suddenly that portion of my yard greened up immensely, having previously
been dead and dry during the dry months.  Most of that area gets covered
year-round with wild strawberry plants, that showed up by themselves.
Before the swales, that was just barren dirt.

A swale's theory of operation is that water that otherwise would flow
over and through the soil of a patch of ground, as runoff, instead is
forced to slow down and spread sideways and down, an underground
reservoir.  It definitely works.  Sometimes, perhaps, too well.