[conspire] guido: more/remaining filesystem corruption on at least root (/) filesytem 8-O

[Michael Paoli]

Rick (& on-list),

Alas, looks like we didn't nail all the filesystem corruption on guido,
or some has come back, at least on the root (/) filesystem.  8-O :-/
I just stumbled across and couldn't help notice ... well, first the
content, then ...:
$ ls -l /etc/debian_version
-rw-r--r-- 0 root root 781 Nov 23 01:42 /etc/debian_version
$
Note the link count of zero!!!, and for content we have:
$ cat /etc/debian_version
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
   "/var/log/libvirt/**/linuxmafia.log" w,
   "/var/lib/libvirt/qemu/domain-linuxmafia/monitor.sock" rw,
   "/var/lib/libvirt/qemu/domain-1-linuxmafia/*" rw,
   "/run/libvirt/**/linuxmafia.pid" rwk,
   "/run/libvirt/**/*.tunnelmigrate.dest.linuxmafia" rw,
   "/var/local/VMs/storage/linuxmafia/sda" rk,
   # don't audit writes to readonly files
   deny "/var/local/VMs/storage/linuxmafia/sda" w,
   "/var/local/VMs/storage/linuxmafia/sdb" rwk,
   "/var/local/VMs/storage/linuxmafia/sdc" rwk,
   "/var/lib/libvirt/qemu/domain-1-linuxmafia/{,**}" rwk,
   "/var/lib/libvirt/qemu/channel/target/domain-1-linuxmafia/{,**}" rwk,
   "/var/lib/libvirt/qemu/domain-1-linuxmafia/master-key.aes" rwk,
   "/dev/net/tun" rwk,
   "/dev/net/tun" rwk,
$
In any case, things are definitely still(/again?) messed up there on the
root (/) filesystem.  :-/

So ... I was at first thinking (before I noticed the link count!) maybe
if it's not "too bad" could fix (e.g. slight shuffling of files) while
it's mounted, and remotely, of the root (/) filesystem ... but there's
no out-of-band management or the like ... and seeing that link count,
and if it's rw and mounted and in use as root ... yeah, probably not a
good idea - most any changes could make it worse - unintended
consequences 'n all that.  So ... probably wait 'till you're reasonably
conveniently available on-site before proceeding any further - at least
on any intentional write changes to the root (/) filesystem itself.

Meantime, since there don't seem to be any issues on /var filesystem
(nor did we earlier see issues there), I'll probably go ahead and snag a
backup image of the root (/) filesystem, etc.  (so worst case would at
least have reference image copy as it currently is) etc. to somewhere on
/var filesystem - similar to what we did earlier in fixing the earlier
bits of mess we found on root (/) and /boot filesystems.  I may put it
somewhere other than under /var/tmp/ - look for something obvious under
/var/ ... since updating ~root/..., well, see below:

Alas, even ~root is on the root (/) filesystem - so that makes even
changes/additions to ~root/Changelog potentially hazardous.
And, peeking at /proc/mounts, kernel hasn't yet noticed or
picked up the corruption and remounted root (/) ro.

Also, given what shows as content of /etc/debian_version
presently, not sure VM(s) would come up fine if, e.g.
guido were to be rebooted - but data of the VMs (on /var)
appear okay, so "worst case" that should be a quite fixable
issue should it occur (recall how we bumped into link state of
VM's interface having changed earlier?  In the configuration?
Yeah, that also shouldn't have happened either) - in any case, as I say,
any such issues that might appear there, should be quite fixable - so
long as the data of the VM itself remains okay - and that part continues
to look fine.

So, well, I guess so far the /relatively/ good news:

As far as I'm aware, no issues with linuxmafia(.com)
and its data (other than the occasional kernel Oops)
and it still continues to be up and running thus far.

Thus far seems to only be root (/) filesystem - I'll have a careful more
thorough look and see what I find ... but basically look don't
"touch"/change, at this point.

Anyway ... followup - mostly off-list (keep the "chatter" down),
and, well, have a peek under /var and see what I leave there ...
may also relocate some/"all" of our earlier /var/tmp/... content to
elsewhere on /var for safe(r) keeping. Will probably stash somewhere
under /var/local/.

And, in the famous words of Douglas Adams:
DON'T PANIC
Alas, sometimes kernel needs/best take exception to that ... but
hopefully generally not too much of that either.