[conspire] Church of Get It Done Now ... linuxmafia.com --> virtual? :-)

[Michael Paoli]

> From: "Rick Moen" <rick at linuxmafia.com>
> Subject: Re: [sf-lug] SFLUG.org
> Date: Fri, 12 Apr 2019 14:27:58 -0700

> (I'll be totally frank:  Although Michael and I chat enough that I can
> in good conscience summarise what he's said wtihout fear of getting it
> wrong, we often have a difference in style:  Michael will want to delay
> more than I do in order to attend to many small details, in situations
> where I'm of the Church of Get It Done Now, and Fix the Small Things
> Later.  And I'm far from claiming that either of those general appraches
> is better than the other.)

Rick,

So ... how's it comin' along with setting up Linux on the spiffy newer
host hardware (SSD(s), fanless, much more modern, ... if I recall
correctly)?  And, somewhere along with that, converting linuxmafia.com
physical host to virtual upon the much newer hardware?  :-)

Ah yes, much easier to dispense good advice than (always and/or
consistently) follow it ... yeah, me too.  8-O

Anyway, was kind'a curious how that's comin' along.  I do know you have
a very careful and well laid out design and architecture for the OS on
the physical host.  Are you waiting maybe for (more) convenient
opportunity/time?  E.g., if not Debian, I don't recall(/know?) details
of your current preferred OS/distro ... but if it is, or is closely tied
to timing of Debian major releases, perhaps you're waiting for Debian's
not so horribly far off in the future now (maybe July?  Debian (mostly)
doesn't have schedules ... but does have projections ... it's released
(stable) when it's darn good and ready), as opposed to having done all
the install/configuration work, and then be looking, after that, at a
major OS version upgrade in the semi-nearish future, as opposed to
likely years or more into the future; ... or ... maybe you'll do
something that (is or) derives from Debian testing?  Anyway, curious the
[gu]estimated timeline on that project.  Maybe you'd like a bunch 'o
SF-LUG and/or CABAL folks to, er, "assist" ... though that could go
sideways.  ;-)

Also, I was thinkin' ... as I recall ... linuxmafia.com (host & config
thereof ... I've actually got access & could peek if I needed to verify
... hmmm...
http://linuxmafia.com/pipermail/conspire/2019-May/009819.html
)
... two drives - as I recall, spinning rust, ... hopefully both still
working at present (I've not looked in a while (okay, let's pretend I
didn't peek), but I think that was and is status and has been for some
fair while?), and with "all" - or at least most all 'o that (all the
partitions, anyway?) md RAID-1 mirrored between the two drives ... or at
least all (or most all?) the "important" stuff.  (And backups too, but
that's a (mostly) separate topic).  So, I was thinkin' ... getting that
from where it is - to the new VM infrastructure and host for it - when
it's ready for that at least, and how to do so, with as little
disruption as feasible to linuxmafia.com (at least the host image).

So, on the existing, and presumably before doing the transfer
"for real", could do test runs - with only a few minor changes for the
"for real" transfer/migration.  Something approximately like this:
o prepare the VM (if not already done) - don't need a whole lot there,
   it can be the [virtual] equivalent of bare metal to install onto - as
   we already have existing host to migrate to it.
   o *DO* match Ethernet MAC address(es) from old to new ... at least
     that's my recommendation (prevent image from glitching over change
     in MAC address) ... and if so, note in one's log that such was done
     (and probably also why) ... or don't, and just use the VM generated
     default MAC - if so, be sure to test that ("dry run") and work out
     any kinks;
     using the VM's default MAC has the advantage that the MAC more
     properly identifies the [virtual] hardware.
   o RAM - not less than that of source host (optionally more, or can
     always increase later)
   o CPU - can match to source host as close as feasible ... or not.
     Default should also work fine, and be exceedingly backwards
     compatible.  Can also always [virtually] "upgrade" that later.
     Also to consider, if you'll want to be able to do migrations of the
     VM between physical, and especially live migrations, you'll want the
     *virtual* CPUs to match - so if the physical is different, you'll
     want least common denominator between the two (otherwise live
     migrations may fail to go from the more capable [virtual] CPU to the
     less capable [virtual] CPU.  (I had to do some very minor virtual
     CPU tweaks to allow the balug VM to be able to do live migrations
     between the "vicki" physical host and my (egad - but quiet!)
     personal laptop).
   o drive(s) - only need one data drive (can add more later - even hot
     add!), if I recall correctly, the "old" drives are (nearly?) the
     same size, make the [virtual] drive not smaller than the
     larger(/largest?) of the "old" drives.
   o most 'o the rest is details and will have reasonable defaults.
     You'll probably want the network setup in "bridged" mode,
     so the [virtual] host will be like it's actually [virtually] on the
     same physical [virtual] LAN.  Can optionally add some firewalling of
     the VM - but totally optional, and that would give LAN more
     protection from the VM than it had before - can also always add that
     later.  There are other ways to do the network infrastructure, but
     given all the servers and stuff needing to be Internet address
     addressable/reachable, likely simplest and (about) best to do
     bridged.  As for bridge, could use basic bridge utilities; or could
     go more snazzy(/complex) with software defined networking and fully
     configurable [virtual] switch, etc.  If the physical host isn't
     already set up via bridge, may need to do that first (i.e. IP of
     physical on bridge device, rather than network port device, etc.)
o verify md RAID-1 status on disks - make sure that's all mirrored (or
   at least all that's to be migrated)
o verify existing disk health reasonably - e.g. see that it can at least
   read end-to end on all disks with 0 errors, e.g.:
   # (for d in /dev/sd?; do dd if="$d" of=/dev/null; done)
o for the "real" run, but not test dry run(s), shut down various
   services, most notably so stuff doesn't otherwise change between start
   of copy and firing up VM as replacement and on-line and atop new
   hardware right after taking old fully off-line (e.g. disconnect
   network cable):
   o MTA - all mail send/receive (avoid duplicates or loss)
   o other services if one doesn't want log or data changes (that
     wouldn't make it to new after copy), e.g.: web, rsync, sshd, ...
o "break" the mirrors - for one of the two drives (probably/preferably
   breaking off the mirror copies on the non-primary (not default boot)
   drive, while leaving the ones on the primary boot live
o make image copies of the following from "old" to new:
   o all the drive data before the first partition (and heck, even into
     first partition) - this can be taken from the primary drive if
     it matches location (offset) of start of 1st partition - copy that
     to the 1st/primary virtual disk of the new VM (can also create a
     backup/reference copy at same time)
   o duplicate partitioning from 2nd drive of old to 1st of new (new can
     start out with only 1 drive)
     # sfdisk -d -uS ...
     # sfdisk -uS ...
     can be very handy for that
   o each of the partitions, copy from old 2nd drive to 1st of new
     again, can optionally also make backup reference copy (e.g. could
     even write both at same time - SSD is fast, spinning rust and/or
     network would be the bottleneck, so wouldn't cost (machine/clock)
     time to make an extra copy at same time (and could even later
     copy that reference copy to separate drive/media - which might be
     slower, but wouldn't slow down the earlier copying)
     Now, since the filesystems weren't cleanly shut down, what's on
     those mirrors won't be "clean", but, since we broke mirrors, it'd be
     rather like we'd just pulled the plug, and they should be quite
     clean "enough" to well recover ...
   o Now we're almost ready to boot new, but not quite - a few more steps
     first
   o boot the VM from recovery media, e.g. [virtual] CD-ROM or USB
     since we "broke" the mirrors, and copied that stuff, they'll be in
     "failed" state - want to use mdadm to take them out out failed
     state
   o o for "dry run": make sure the virtual won't go on-network (e.g.
       disable its virtual network connectivity - do the equivalent of
       [virtually] pulling the network cable
     o for the "for real" cutover: pull network cable (or shutdown) old,
       be sure network connectivity ([virtual] cable) is enabled on new,
       boot new - filesystems will start off unclean, but should recover
       very quickly (if journaled, e.g. ext[34]), or in some moderate bit
       (depending upon filesystem size, etc.) if not journaled (e.g.
       ext2).
   o At that point, "for real" (or dry run) migration is mostly done.
     Then just a bit of clean-up:
     o in the "for real", or dummy run with network [virtually]
       unplugged, reenable the network services that were disabled (e.g.
       MTA) - and they should pickup quite cleanly from the point where
       the mirrors were broken on "old" source host
     o md will be complaining about the RAID-1 stuff missing the mirror
       devices (nominally expecting 2 each, only 1 each present).
       Can either:
       o reconfigure the RAID-1 devices to only have *one* device each
         ("fake" RAID-1 - but also easy to change back to proper RAID-1
         later, if/when that may be desired)
       or:
       o add the missing devices (e.g. use/add 2nd [virtual] drive).

I don't think I missed any (major?) steps.  "Of course" that's not the
only way to migrate - but does keep service interruption time pretty
low (mostly just copy time + boots & some minor config/service tweaks).

And "of course", once linuxmafia.com is all running on virtual, quite
easy/handy to "clone"/backup that, do dry run experiments on it, e.g.
how [not] painful to upgrade to current Debian stable, cross-grade to
64-bit, whatever (if I do X, will it break Y?), investigate potential
migration to clean new OS install (on virtual), etc.