[conspire] A little out here, a little out there (linuxmafia.com, balug, ...)

[Michael Paoli]

A little out here, a little out there (linuxmafia.com, balug, ...)
So, guessing power disruptions or whatever,
anyway, looks like guido had reboot(s) yesterday (Sunday):
$ hostname --fqdn &&
> TZ=US/Pacific stat  --printf='%x %n\n%y %n\n%z %n\n' /proc/1 | sort -u
guido.linuxmafia.com
2024-02-04 14:36:38.304000319 -0800 /proc/1
$
And of course then also, linuxmafia.com (the linuxmafia VM thereupon),
though looks like some more recent there:
$ hostname --fqdn &&
> TZ=US/Pacific stat  --printf='%x %n\n%y %n\n%z %n\n' /proc/1 | sort -u
linuxmafia.com
2024-02-05 09:38:17.581599151 -0800 /proc/1
$
Anyway, looks like guido lost the mirroring somewhere along the way,
perhaps more than just PG&E sneezed on or looked funny at flakey
power connections:
$ hostname --fqdn && cat /proc/mdstat | sed -e '/^ *$/d'
guido.linuxmafia.com
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md6 : active raid1 sda6[2]
       19512320 blocks super 1.2 [2/1] [_U]
md8 : active raid1 sda8[2]
       95171584 blocks super 1.2 [2/1] [_U]
md7 : active raid1 sda7[2]
       7805952 blocks super 1.2 [2/1] [_U]
md1 : active raid1 sda1[2]
       497664 blocks super 1.2 [2/1] [_U]
md5 : active raid1 sda5[2]
       1949696 blocks super 1.2 [2/1] [_U]
unused devices: <none>
$
Looks like one of the drives wasn't seen at boot:
$ ls -d /dev/sd[ab]; (cd /sys/block && grep . sd[ab]/size)
/dev/sda  /dev/sdb
sda/size:250069680
sdb/size:3907029168
$
But after a rescan:
# (for tmp in /sys/class/scsi_host/host*/scan; do echo '- - -' >> 
"$tmp"; done)
# dmesg | tail -n 19
[72980.222743] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[72980.223688] ata3.00: ATA-9: Samsung SSD 840 PRO Series, DXM04B0Q, max 
UDMA/133
[72980.223703] ata3.00: 250069680 sectors, multi 16: LBA48
[72980.224107] ata3.00: Features: Dev-Sleep
[72980.225382] ata3.00: configured for UDMA/133
[72980.235994] scsi 2:0:0:0: Direct-Access     ATA      Samsung SSD 840  
4B0Q PQ: 0 ANSI: 5
[72980.236637] ata3.00: Enabling discard_zeroes_data
[72980.236666] sd 2:0:0:0: [sdc] 250069680 512-byte logical blocks: (128 
GB/119 GiB)
[72980.236840] sd 2:0:0:0: [sdc] Write Protect is off
[72980.236846] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[72980.236886] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[72980.236936] sd 2:0:0:0: [sdc] Preferred minimum I/O size 512 bytes
[72980.237550] sd 2:0:0:0: Attached scsi generic sg2 type 0
[72980.237777] ata3.00: Enabling discard_zeroes_data
[72980.238793]  sdc: sdc1 sdc2 < sdc5 sdc6 sdc7 sdc8 >
[72980.239371] sd 2:0:0:0: [sdc] Attached SCSI removable disk
[72980.550831] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[72980.553492] ata4.00: configured for UDMA/133
[72980.564301] ata4.00: Enabling discard_zeroes_data
#
Then the 2nd eSATA attached drive is seen fine:
$ ls -d /dev/sd[ac]; (cd /sys/block && grep . sd[ac]/size)
/dev/sda  /dev/sdc
sda/size:250069680
sdc/size:250069680
$
Added back the missing mirrors:
# (set -x; for n in 1 5 6 7 8; do mdadm --manage /dev/md"$n" --add 
/dev/sdc"$n"; done)
+ for n in 1 5 6 7 8
+ mdadm --manage /dev/md1 --add /dev/sdc1
mdadm: added /dev/sdc1
+ for n in 1 5 6 7 8
+ mdadm --manage /dev/md5 --add /dev/sdc5
mdadm: added /dev/sdc5
+ for n in 1 5 6 7 8
+ mdadm --manage /dev/md6 --add /dev/sdc6
mdadm: added /dev/sdc6
+ for n in 1 5 6 7 8
+ mdadm --manage /dev/md7 --add /dev/sdc7
mdadm: added /dev/sdc7
+ for n in 1 5 6 7 8
+ mdadm --manage /dev/md8 --add /dev/sdc8
mdadm: added /dev/sdc8
#
And bit later, all properly mirrored again:
$ cat /proc/mdstat | sed -e '/^ *$/d'
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md6 : active raid1 sdc6[3] sda6[2]
       19512320 blocks super 1.2 [2/2] [UU]
md8 : active raid1 sdc8[3] sda8[2]
       95171584 blocks super 1.2 [2/2] [UU]
md7 : active raid1 sdc7[3] sda7[2]
       7805952 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdc1[3] sda1[2]
       497664 blocks super 1.2 [2/2] [UU]
md5 : active raid1 sdc5[3] sda5[2]
       1949696 blocks super 1.2 [2/2] [UU]
unused devices: <none>
$

Meanwhile, other side of the bay, Sunday, well over half a dozen (short,
but nevertheless) power outages / dropouts, so hit the balug VM a fair
bit (hosts BALUG.org, SF-LUG.org, BerkeleyLUG.com, ...).
After a couple rounds or so of that, I live migrated it to vicki (ye
olde loud power hungry machine) where at least there it can go from 
power
up to fully booted without any human intervention).  I'll likely migrate
it back again when the power situation seems at least meta-stable again
(maybe it's to that point by about now or so?).
$