[conspire] linuxmafia.com disk drives & (md) RAID-1 ... uh oh ?
Michael Paoli
Michael.Paoli at cal.berkeley.edu
Thu May 23 22:47:14 PDT 2019
Yeah, ... given situation of the hardware, and how the
disk layouts exist on there, and (near?) future migration onto
the CompuLab hardware, etc., and (sufficiently?) regular
backups, I'm inclined to agree - spend the spare cycles
(more) on the CompuLab, and not (or less) on ye olde
hardware and disk and RAID situation there.
In a wee bit more detail (for the curious or whatever) ...
In addition to what Rick noted on the hardware situation and its
relative precariousness ...
Two functioning drives - /dev/sd{a,b}
sda significantly larger, md RAID stuff is on sdb - at present the
mirrors missing, so it's degraded RAID-1, mirrors were on (apparently)
3rd drive that (quite) matched the 2nd (current sdb) drive.
No md RAID stuff presently on sda. Though sda the larger, no simple
way to mirror the existing degraded RAID-1 from sdb to sda - space on
sda is mostly more-or-less used up - at least partitions, etc. ... even
if some might not be in active use - in any case, not sufficient free
space on partitioning to add partitions to mirror what's presently got
missing mirrors - at least short of much more involved stuff like
shrinking filesystems, repartitioning, reboots, etc., etc.
Also, though much is configured as RAID-1, a fair bit isn't, so
fixing the RAID-1 stuff so it's not degraded, would cover more,
but also certainly not everything.
In short, non-trivial work - if even feasible - and not so much to gain.
And, for those interested in more detail, here's (at least most of) the
relevant data I looked at on the situation:
2019-05-24T05:01:26+00:00
$ hostname && (cd /sys/block && grep . sd[a-z]/device/{model,vendor}) | sort
linuxmafia.com
sda/device/model:IC35L073UWDY10-0
sda/device/vendor:IBM
sdb/device/model:ATLAS_V_18_WLS
sdb/device/vendor:QUANTUM
$ mount -t ext2,ext3,ext4 | awk '{print $5,$3,$1;}' | sort
ext2 /boot /dev/sda1
ext2 /usr /dev/sda9
ext2 /var /dev/sda6
ext3 / /dev/sda7
ext3 /home /dev/md3
ext3 /recovery /dev/sda8
ext3 /usr/local /dev/md4
ext3 /var/lib /dev/md1
ext3 /var/spool /dev/md2
ext3 /var/www /dev/md0
$ grep . /sys/block/sd[ab]/size
/sys/block/sda/size:143374805
/sys/block/sdb/size:35861388
$ df | egrep -v '^(tmpfs|udev) '
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda7 918322 251869 617457 29% /
/dev/sda1 180639 28690 142312 17% /boot
/dev/md3 5763508 3103864 2366868 57% /home
/dev/sda8 918322 16463 852863 2% /recovery
/dev/sda9 4807056 1294084 3268788 29% /usr
/dev/md4 5146912 2700540 2184920 56% /usr/local
/dev/sda6 5763616 789828 4681008 15% /var
/dev/md1 1369855 1231855 64912 95% /var/lib
/dev/md2 2885664 425752 2313328 16% /var/spool
/dev/md0 1829037 633133 1098317 37% /var/www
$
# sfdisk -uS -l /dev/sda
Disk /dev/sda: 8924 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/sda1 * 63 385559 385497 83 Linux
/dev/sda2 385560 26764289 26378730 5 Extended
/dev/sda3 26764290 143364059 116599770 83 Linux
/dev/sda4 0 - 0 0 Empty
/dev/sda5 385623 1365524 979902 82 Linux swap / Solaris
/dev/sda6 1365588 13076909 11711322 83 Linux
/dev/sda7 13076973 15036839 1959867 83 Linux
/dev/sda8 15036903 16996769 1959867 83 Linux
/dev/sda9 16996833 26764289 9767457 83 Linux
# sfdisk -uS -l /dev/sdb
Disk /dev/sdb: 2232 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/sdb1 63 35841014 35840952 5 Extended
/dev/sdb2 0 - 0 0 Empty
/dev/sdb3 0 - 0 0 Empty
/dev/sdb4 0 - 0 0 Empty
/dev/sdb5 126 3903794 3903669 fd Linux raid autodetect
/dev/sdb6 3903858 6827624 2923767 fd Linux raid autodetect
/dev/sdb7 6827688 12691349 5863662 fd Linux raid autodetect
/dev/sdb8 12691413 13671314 979902 82 Linux swap / Solaris
/dev/sdb9 13671378 25382699 11711322 fd Linux raid autodetect
/dev/sdb10 25382763 35841014 10458252 fd Linux raid autodetect
# blkid /dev/sda[135-9] /dev/sdb[15-9] /dev/sdb10
/dev/sda1: LABEL="/boot" UUID="96ce6da4-7b55-409e-a078-3572612b61c1"
TYPE="ext2"
/dev/sda5: TYPE="swap" UUID="dded258a-f6ce-4a4c-9ee7-5d8076648080"
/dev/sda6: LABEL="/var" UUID="728b737e-8420-437c-a43a-5d5a8f60fba5"
TYPE="ext2"
/dev/sda7: LABEL="/" UUID="0dc3dcff-3d63-4830-893f-6f9afd811875"
TYPE="ext3" SEC_TYPE="ext2"
/dev/sda8: LABEL="/recovery"
UUID="ed5d4608-6db8-40c0-9405-ba091b5f8a77" TYPE="ext3" SEC_TYPE="ext2"
/dev/sda9: LABEL="/usr" UUID="5f702fae-9386-436e-86d2-90323b7f0857"
TYPE="ext2"
/dev/sdb5: UUID="66f7d6e3-fa93-8e04-7e7c-38d0deadc6c4"
TYPE="linux_raid_member"
/dev/sdb6: UUID="0099f081-e6fb-e0d8-29ee-eeda14fbe85f"
TYPE="linux_raid_member"
/dev/sdb7: UUID="c73530f1-61e1-9518-71be-9c1f74071e94"
TYPE="linux_raid_member"
/dev/sdb9: UUID="fd756273-33ae-ddec-1b14-da158b5a081e"
TYPE="linux_raid_member"
/dev/sdb10: UUID="54924f26-e9fe-e20a-9f93-577242a3093d"
TYPE="linux_raid_member"
# cat /proc/swaps
Filename Type Size Used
Priority
/dev/sda5 partition 489940 118704 -1
# mdadm --detail /dev/md?* 2>&1 | egrep '^/dev/|active'
/dev/md0:
0 8 21 0 active sync /dev/sdb5
/dev/md1:
0 8 22 0 active sync /dev/sdb6
/dev/md2:
0 8 23 0 active sync /dev/sdb7
/dev/md3:
0 8 25 0 active sync /dev/sdb9
/dev/md4:
0 8 26 0 active sync /dev/sdb10
#
> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
> Subject: Re: [conspire] linuxmafia.com disk drives & (md) RAID-1 ...
> uh oh ; -}
> Date: Thu, 23 May 2019 21:50:17 -0700
> Hmmm, I'd forgotten some fair bit of those hardware details (most
> specifically what types of buss(es) on the hard drives, what the
> initial situation was when we'd earlier worked on it to get
> linuxmafia.com back up and on-line again, and exactly what
> situation we left that in hardware-wise).
>
> So, yeah, I certainly well see your points.
> And at least with sufficiently regular backups, I'm inclined
> to agree - don't want to add drive(s) to the existing ...
> especially that put significantly more load on PSU (e.g.
> more spinning rust).
>
> Maybe I'll have a look around wee bit (been a while) - see if I might
> have any feasible recommendations, notably regarding the data, and
> mirroring, etc. ... without changing the hardware ... heck, given
> the PSU & hardware vintage and relative scarcity of stock of good
> reliable replacement parts - probably even best to avoid power
> cycling the thing - at least as feasible.
>
> And I'm presuming also disk I/O performance isn't critical -
> e.g. if disk specs/busses don't match, that's not critical,
> so long as nothings (way) too slow/unreliable - e.g. native
> busses okay, USB or the like - for backup sure, but not for
> RAID or active filesystem use (e.g. not OS or any significantly
> regularly used data).
>
> And ... 18G goes quite a long way on a modest installation.
> The balug VM ...
> $ echo 'b='"$(cat
> /sys/block/sda/size)"';b*512;b*512/1000000000;b*512/1024/1024/1024'
> | bc -l
> 17179869184
> 17.17986918400000000000
> 16.00000000000000000000
> $
> A mere 16 GiB.
> That also means, on suitable physical hardware and newer larger drives,
> (and RAM), can have a whole lots 'o VMs there. :-)
>
>> From: "Rick Moen" <rick at linuxmafia.com>
>> Subject: Re: [conspire] linuxmafia.com disk drives & (md) RAID-1
>> ... uh oh ; -}
>> Date: Thu, 23 May 2019 21:22:10 -0700
>
>> Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
>>
>>> Hmmm, I thought you(/we/I) had earlier fixed that?
>>
>> No, I'm kind of caught between the devil and the deep blue sea.
>> Here's the dilemma, and feel welcome to tell me that my objective of
>> caution would be better served by making the opposite decision:
>>
>> As you may recall, the qty. three single-ended SCSI hard drives were
>> pulled from my very last VA Linux model 2230 when its very last Intel
>> L440GX+ 'Lancewood' PIII suddenly died.
>>
>> Unresolved question: What killed the motherboard? What killed some of
>> its identical predecessors, across a number of different supposedly
>> identical VA Linux chassises and PSUs? In fairness, I got extremely
>> long service out of my leftover VA Linux gear, and probably ought to
>> just not sweat that question. But it lingers in the background, as to
>> the rest of this.
>>
>> When that happened, before resorting to biting the bullet and doing a
>> total do-over, you and I tried a thing. Although I had no more spare
>> Lancewood, I did have one non-Lancewood PIII box, that old fleabitten
>> and dubious-looking Rackspace 2U. So, on a what-the-hell-let's-see
>> basis, we had the idea of seeing if the Rackspace would boot my SCSI
>> drives.
>>
>> The SCSI drives were:
>> o one 73GB drive
>> o a pair of 18GB drives operated RAID1 using MDraid
>>
>> Yes, really. _That_ small, because we're talking about scrounged
>> hardware that was already obsolete when I scrounged it (like, at the
>> time, picking up used server-class SCSI hard drives for $5 each because
>> you've been in survival mode after getting laid off by VA Software
>> Corporation right into the opening crescendoes of the Dot-Bomb tech
>> depression -- thank you, Larry!), and in addition, it's now scary-old.
>>
>> So, there we were, pondering shotgun-marrying my hard drives to the
>> dubious server, and I was hairy-eyeballing it's particularly dubious
>> PSU. And thinking, urp? What's the nastiest SPoF hardware risk on a
>> typical server? Answer: The PSU. And how do you mitigate that risk?
>> Answer: You assume the PSU is crap and replace it with one that you
>> know is not crap.
>>
>> But, for various reasons, this was not done at that time, so caution
>> lead me to think: Assume a weak PSU. (Noted in passing: Notoriously,
>> when PSUs fail, e.g. because you tried to draw more current than they
>> are able to reliably serve, they show an uncanny instinct for burning
>> out any and all attached hard drives as they spike out.)
>>
>> OK, so weak PSU. The way you baby along a weak PSU is to not burden it
>> with any more attached electronics (drawing current) than absolutely
>> necessary. I stared at the PSU, I stared at the three drives, and I
>> thought: The smallest possible electrical load while still test-booting
>> all of my filesystems would be achieved by omitting one of the 18GB
>> RAID1 drives. So, that's what we did, connecting two of the three
>> drives, omitting half of the RAID1 pair.
>>
>> And it booted.
>>
>> I did a little dance of joy. The server was back live. In due course,
>> I made triple-sure that I absolutely had good and complete backups, and
>> have continued to do so, periodicaly, ever since.
>>
>> At any given time, I _could_ choose to spin down the ratty Rackspace
>> box, cable in the third drive's data connector and the Molex power
>> connector, rub my lucky rabbit's foot, take a deep breath, and power the
>> whole assemblate of string and chewing gum back _up_.
>>
>> I could do that. But I'd feel really stupid if, within a day or two,
>> the unit let its magic smoke out and fried the attached hard drives
>> because the crap PSU could drive two enterprise-class SCSI drives but
>> not three of them.
>>
>> I'm unconvinced that Rackspace ever intended the small El Crappo case to
>> house three SCSI drives, and therefore uncertain the dubious PSU can
>> reliably power that many. I've seen a lot of 2U designs that _are_
>> intended for three such drives, and this thing doesn't look like it has
>> the grunt.
>>
>> Through today, I've guesstimated that the risk of that degraded-RAID
>> hard drive failing is less severe than the risk of three drives
>> destroying everything by clobbering the PSU and the PSU shooting
>> everything else in the foot. Consider: If the half-RAID drive dies,
>> that's annoying, but recovery is then easy and obvious: I remove the
>> failed drive, replace it with the long-estranged other drive, boot the
>> system back up, and update the half-RAID partitions' file contents from
>> backup. Done. All I lose is an hour of fiddling and changes since my
>> most recent backup. Provided I do frequent backup-updating, the
>> loss-exposure is very small and not worth worrying about.
>>
>> I do without RAID1 continuity-of-operation in the face of _some_ hard
>> drive failures. What I gain is substantially reduced risk of total
>> electronics destruction from an overstressed PSU.
>>
>> Which choice do _you_ think better?
>>
>>
>> Personally, I'm disinclined to do hardware changes other than
>> decommissioning the drives, the box, the whole kit'n'caboodle,
>> Because my watch says it's not 1997.
>>
>>
>>
>>> Are you in need of drive?
>>
>> Well, yes and no, but mostly no.
>>
>> I actually own a lot of unused hard drives. Many of them are
>> spectacularly better than those three. But that's asking the wrong
>> question, really.
>>
>> The optimal target (among those in my possession) is the pair of Samsung
>> 128GB SSDs I bought for the CompuLab IntensePC. I just have to get
>> there. Getting there is complicated. Not a complaint, but time talking
>> about the matter on mailing lists doesn't make that happen sooner, and
>> arguably does the opposite.
>>
>>
>>
>>
>>> Do I recall correctly it uses [P]ATA/IDE?
>>
>> So close, and so wrong. ;->
>>
>> Back then, before the convergence of SAS with SATA, I'd have died of
>> shame if I'd ever been guilty of building a server on PATA. I used to
>> work at VA Linux Systems, not Bob's Toy Shop, Pet Obedience School, and
>> Pet Taxidermy Service.
>>
>>
>>> Oh, and CABAL, Saturday ... I'll probably be there (thus far intending
>>> to).
>>
>> Terrific! I'm still pondering the what-to-cook thing. Maybe pizza
>> again, as that's a sure-fire.
>>
>>> Do also have hardware if you might be interested (even picked up some
>>> more today I probably need to add to list):
>>> https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware_etc
>>> Holler if there's something offered there I have that you want (and may
>>> want me to bring to Cabal)
>>
>> Thanks. As the kids say, 'I'm good.'
>>
>>> Anyway, let me know if you need/want assistance, or want me to look
>>> into the md situation further (and/or correct if feasible).
>>
>> Probably more useful to spend initial time on the CompuLab.
More information about the conspire
mailing list