[sf-lug] details: Re: online again now: Re: outage: [www.]{sf-lug, balug}.org

[Michael Paoli]

And, for those wanting the details:

https://lists.balug.org/pipermail/balug-admin/2018-November/000972.html
----- Forwarded message from Michael.Paoli at cal.berkeley.edu -----
     Date: Wed, 21 Nov 2018 16:59:41 -0800
     From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
  Subject: online again: outage: [www.]{sf-lug,balug}.org
       To: BALUG-Admin <balug-admin at lists.balug.org>

online again,
the short version of the outage synopsis:
unmanaged switch failed, connectivity lost around:
2018-11-21T02:06:39Z
switch replaced, connectivity restored around:
2018-11-21T23:42:09Z

the longer version, first mostly from the log, and with some adjustments
on details:
2018-11-20T-08:00
2018-11-21T+00:00
Lost (Internet+) connectivity, apparently first briefly, then solid:
Nov 21 01:44:09 [physical host that has BALUG as guest VM] kernel:  
[3213467.919511] e1000e: eth0 NIC Link is Down
Nov 21 01:44:41 [physical host that has BALUG as guest VM] kernel:  
[3213499.958862] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex,  
Flow Control: Rx/Tx
Nov 21 02:06:39 [physical host that has BALUG as guest VM] kernel:  
[3214817.086977] e1000e: eth0 NIC Link is Down
POTS connection verified still good; deferred remaining troubleshooting, etc.

2018-11-21
Investigated connectivity issue:
tigger shows link down - both software and hardware LEDs,
unmanaged gig switch:
D-Link MODEL: DGS-1005D P/N: BDGS1005D.B1 S/N: DR1914C001239 H/W Ver.: B1
shows no link or activity (host tigger connected to Internet via above),
only LED(s) lit on it are power.
Westell DSL modem also shows no link nor activity, otherwise appears
normal ("top" two LEDs, "Power" and "Ready" are both solid green),
suspected switch, but covering bases also,
tried power cycling the Westell DSL modem - since it tends to require
that once in a while anyway (but fails in different mode than current
situation) ... no change.
Tried substituting in another DSL modem - probably not needed, but given the
semiregular failures of the Westell, thought might be good to try out
other (a Motorola) DSL modem for a while and see if it's more reliable
... well, that one worked some year(s) back, but seems to have subsequently
failed and no longer work as far as I could easily and quickly tell -
it showed absolutely no signs of life (not a single LED lit),
so switched back to the Westell again.
After powercycling the switch, not even the power LED on switch would
come back on (though it may have briefly flickered), and still couldn't
get it to show any link on any ports.
Swapped out switch, removing the:
D-Link MODEL: DGS-1005D P/N: BDGS1005D.B1 S/N: DR1914C001239 H/W Ver.: B1
and replacing it with:
TP-Link Model: TL-SG1008D S/N: 2177442009713 8-Port Gigabit Desktop  
Switch (unmanaged), and all fine again after that, service restored at:
Nov 21 23:42:09 [physical host that has BALUG as guest VM] kernel:  
[3292541.509278] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex,  
Flow Control: Rx/Tx

Anyway, ... guess that
D-Link MODEL: DGS-1005D P/N: BDGS1005D.B1 S/N: DR1914C001239 H/W Ver.: B1
it had zero issues prior to this failure, and I originally obtained it
and had put it into service ... from the log:
2005-10-17
purchased and installed:
D-Link
MODEL: DGS-1005D
P/N: BDGS1005D.B1
S/N: DR1914C001239
H/W Ver.: B1
and including power supply:
D-Link
Model No.: JTA0302C
2R01044803741
from CompUSA
replacing/"retiring":
Hawking Technology
Model NO: PN108ES

> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
> Subject: online again now: Re: outage: [www.]{sf-lug,balug}.org
> Date: Wed, 21 Nov 2018 15:48:16 -0800

> Back online again now.
>
> More details a bit later.
>
>> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
>> Subject: outage: [www.]{sf-lug,balug}.org
>> Date: Tue, 20 Nov 2018 18:59:24 -0800
>
>> And ... outage again :-\
>>
>> Guestimating if it's the "usual", will likely be on-line again
>> by sometime Wednesday evening (I may or may not get to it before then).
>>
>> Per earlier:
>> impacts all [*.]sf-lug.{org,com} & [*.]balug.org
>> SF-LUG lists remain up and on-line (at least as far as I'm aware).
>>
>> Also, DNS mostly not impacted (in general, slaves remain functional),
>> though there may be some additional latencies on DNS due to failovers
>> to other nameservers.
>>
>>> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
>>> Subject: "all better now:" Re: outage: [www.]{sf-lug,balug}.org
>>> Date: Wed, 10 Jan 2018 20:15:04 -0800
>>
>>> And ... again, same deal, went off-line at:
>>> <~= 2018-01-11T01:23:21+00:00 2018-01-10T17:23:21-08:00
>>> and back on-line by:
>>>> ~= 2018-01-11T04:04:06+00:00 2018-01-10T20:04:06-08:00
>>> ... again, swift kick to the power switch on DSL modem to reset it,
>>> and "all better" - again ... at least for now.
>>>
>>>> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
>>>> Subject: "all better now:" Re: outage: [www.]{sf-lug,balug}.org  
>>>> (ETR: eveningish)
>>>> Date: Wed, 06 Dec 2017 18:08:09 -0800
>>>
>>>> And ... swift kick to the power switch on DSL modem to reset it,
>>>> and "all better now".
>>>> Looks like the outage started around:
>>>> BALUG PING: ping6 2001:470:1f04:19e::2 FAILED 2017-12-06T20:53:56+00:00
>>>> GW PING: ping 198.144.194.233 FAILED 2017-12-06T20:54:09+00:00
>>>> and service restored a few minutes ago
>>>>
>>>>
>>>>> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
>>>>> Subject: outage: [www.]{sf-lug,balug}.org (ETR: eveningish)
>>>>> Date: Wed, 06 Dec 2017 14:46:38 -0800
>>>>
>>>>> So,
>>>>>
>>>>> They were up and on-line this morning, at least as late as about
>>>>> mid-morning or later, but off-line now.
>>>>> POTS line appears to be working, but not IP,
>>>>> likely the DSL modem needs to be reset (powercycled) again ...
>>>>> but don't have means to do that remotely.
>>>>>
>>>>> Presumably I'll have this resolved sometime this evening once I'm
>>>>> on-site again.
>>>>>
>>>>> impacts all [*.]sf-lug.{org,com} & [*.]balug.org
>>>>> SF-LUG lists remain up and on-line (at least as far as I'm aware).