[conspire] Church of Get It Done Now ... linuxmafia.com --> virtual? :-)

[Rick Moen]

Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):

> <snip *long* (and presumed quite partial(?!) list 'o (other) stuff>

Yep.  Part of the meta-message is that, when I'm the only able-bodied
member of my household and everyone else treats me with often
breathtaking highhandedness as staff, doesn't value my time and
interrupts me constantly, it becomes incredibly difficult to both
allocate a huge block of time for a (possibly somewhat open-ended)
project that requires concentration and considerable care.  There is a
recurring tendency to kick the problem down the road, instead, because
life becomes a circus-spectable of juggling chickens and chainsaws --
and also because instead having some fresh-baked garlic bread in the
shade with a nice glass of red wine while checking on how well the
carrots are growing is immediately rewarding in exactly the way the
other stuff is not.

I used to be a renter, which changed for sorrowful reasons in 2011.
Back when I was a renter, if the faucets leaked or the gutters needed
cleaning or the toilet was backed up, I could blow that off, call the
landlord, and concentrate on finishing the current server migration.

Since 2011, I'm The Man, so I'm a walking wetware trouble-ticket system
for all purposes, no nights off, no escalation path, no additional
manpower.  As I said, hell, nobody _even_ ever empties a wastebasket
except for me.  Or answer the telephone.  Or the front door, usually.

Up until a few years ago, Cheryl _did_ occasionally get the mail from
the mailbox, but that proved to be an utter debacle, with my
retroactively finding piles of unopened misplaced mail including
past-due bills in irrational places in the house.  I had to deploy my
seldom-used powers as sole owner of the house everyone is living in to
insist that, hencefoth, nobody but me is allowed to retrieve, sort, and
distribute the mail for any reason whatsoever, as every other way I
tried to curb the insanity failed.

So, the joys of being a homeowner, living with a pair of only-child
family members who consistently do the 'I forgot' dance about just about
anything and everything, plus a neurotic cat who thankfully hasn't taken
a crap on the Persian carpets in many months and need taking to the
emergency vet in the middle of the night, that's my life on all days
with names ending in 'y'.  And you wonder why the CompuLab server build
remains pending?  There's a clue or two, there.

Cheryl's idea of how to do yard work is to wander around pointing out
problems to me, with the expectation that I'll instantly drop work on
the previous thing she mentioned to do the new one, with no
consideration whatsoever for priorities or arranging work in an order
that makes sense, and _never_ a consideration of writing things down and
doing triage.  No, just interrupt-driven living in the moment is
apparently the recommended not-at-all-a-plan.

Imposing order on that chaos?  Good idea, Rick!  Why don't you add that
to your list of things to do?  You'll get to it all; we have confidence
in your performance.  We'll double your non-existent wages.  Help you?
Don't be silly.  We're telling you all the things you should do; isn't
that enough?





> Well, I certainly may be able to help with that - or certainly at least
> parts of it, and especially so if it's Debian, and maybe up to nearly
> as much if it's some Debian derivative (after all, more Linux distros
> are Debian or derivatives thereof, than any other distro).
> Some bits ...:

One, thanks. 

Two, we're at this point in some mild danger of bikeshedding.
Unfortunately, mailing lists are really dreadful places to do project
management for a number of reasons.  A lot of this is best done
in-person with a couple of legal pads, real-time interactive discussion, 
and triaging punchlists.

Three, I need to ward off input and questions from everyone who is not a
senior sysadmin and highly trusted by me, because this thread is not an
invitation to time-wasting kibbitzing.

The linear and serial nature of mailing list discussion in some contexts
becomes a problem, and I'm 99.9% sure this is one of those times.

Rather than doing point-by-point, I'll provide a big-picture recap.


The CompuLab has a lot of deliberate radical-simplicity design details
that I consider essential but don't want to get into here.  But in broad
picture, it's a fanless, no-moving-parts but beefy Celeron box with
two mirror-pair RAID1-able SSDs on eSATA, and ample RAM for a runtime
operation that has the production host in one VM, and the
next-generation beta of the moment in another.  Given that framing, it's
vital to get the host-OS setup absolutely right the first time, because
it's the SPoF part that's not easily changed and the linchpin of the
whole thing.

Shifting to the _production virthost_ for a moment before returning to
the underlying host OS:

It'll be Lighty or nginx.  Absolutely not Apache httpd any more.
(Leaning towards nginx, with the minor obstacle that I've never
built/run that before, only had encounters with it at workplaces.)

It'll be NSD for auth DNS. Absolutely no BIND9 any more.  And there must
be no even slightly exposed recursive service, which creates the puzzle
about what place(s) to put Unbound.

I'm thinking either OpenNTPd (from OpenBSD) or chrony for NTP, rather
than continuing to use the ISC-sponsored reference implementation, which
I guess we call NTP Project's daemon.  I keep seeing people bitch about
OpenNTP allegedly being deficient because it lacks this-or-that weird
feature that I clearly don't need, which suggests that it's right up my
alley.

Exim4/Eximconfig/SA-Exim again, despite it being unfrashionable, because
fuck that and I'm tired of arguing with people expecting me to justify
why I'm not running Postfix.  I mean, really.  Did they think I've never
spent time on Usenet and learned how to spot that time-wasting
rhetorical gambit?  No, I actually do not need to justify why I'm not
doing X; thanks for playing.  This way to the Great Egress.

You will note the recurring theme that 'Big Dumb Daemons That Do
Everything' get noped.  This reasoning also dictates shitcanning GRUB,
and good riddance.

crond, atd, rsyncd, vsftpd, perfect as they are, carry forward.

Your point from quite a while ago is very well taken, that maybe my
version 1.0 production virthost ought to be a literal copy of the ratty
one running on the Rackspace 2U with the degraded RAID array.  Because
that's the quickest way to get off the scary decrepit hardware.


Back to the host OS, at this point.

I'm still murky on the network setup.  You and I talked about this once
at BerkeleyLUG, which definitely was a start, thanks.  

Kernel will start with a tactically selected Debian source _package_.
No, I'm not mad enough to go totally upstream vanilla kernel.org git
checkout without necessity, thanks for asking.  Some time is worth
spending checking Debianised patchsets for security tightening, whatever
is most satisfactory now that Grsecurity/PaX went off in a snit.  Then,
the real work of making a truly suitable .config, exactly appropriate to
the software,  and tweaking runtime settings to disable unneeded
functionality.  Select best scheduling algorithm.  _Long_ experience
reading CVEs at $FIRM while administering the online credit-card
processing operation made super-clear that the code you simply don't
run, and the features you deliberately disable because you actually
don't need them, are the code and features whose CVEs you can ignore.

Make sure cron job for TRIM is there.  Hard look at mount options.
Filesystem setup is a whole separate discussion, which we touched on at
BerkeleyLUG recently. ext4 will be heavily featured, full stop.
Advocates of anything else, don't waste your breath.  I don't want to
hear 'But why aren't you...?' and have no time for 'explaining'
(justifying) what I'm doing.

Installed packages on the host OS must be very sparse, not a single one
that doesn't absolutely need to be there.  The host OS is intended to be
security-hard and totally minimal.  It's enough Linux to run a kernel
and libc and VMs, monitor them, repair/backup/restore/diagnose them, and
_nothing more_.  Why all this attention to the host OS?  Because unlike
the guest OSes, you cannot just blow away a problem and fix it by
blessing the next-generation beta VM as the new production host.  So, I
must get it right.

'Sparse' includes the miminal software complement for VM that we can
reasonably get away with.  libvirt seems difficult to avoid because its
absence is too painful, but a hard look at other 'recommend/suggests'
packages is in order, because there's a bad history of Debian package
maintainers deciding that bloat is fine by them.

Oh, and by the way, the starting point will not technically be Debian,
but rather Devuan 2.0 ASCII.  I'm still leaning towards OpenRC as the
init system, but would consider s6, even though I'm even less familiar
with it.  Probably OpenRC, to not bikeshed overmuch.  As the late Adam
Osborne of Osbourne Computer used to say, 'Adequacy is sufficient.'

After the host OS and VM setup is peachy, then we take a deep breath,
have another cup of coffee, and ponder the VMs, which is at least easier
because more error-tolerant.  (I mean, I can screw up something and more
easily fix it without downtime.)

But probably even before that, backup/restore should be nailed down, and
other external infrastructure.  I have a couple of Zotac mini-PCs in
reserve, maybe those can help address recursive DNS, and version control
repo needs, and backup, and configuration management ('CM').  Didn't
mention configuration management, did I?  That's going to be an
essential, because, see, I'm embarrassed that at work one makes sure
there are no special-snowflake machines that were hand-built and you're
almost afraid to touch them at all.  The competent regime is to always
be able to re-kickstart (in a CentOS shop) any machine if it has a
problem, at the end of which it load a CM agent which checks in with the
CM host and tells it what additional packages and conffiles it needs to
be a $FOO machine, and so, 15 minutes later, without manual
intervention, you have a spanking-new autobuilt machine.

It's lastingly embarrassing that I've never done this at home because of
the Shoemaker's Children Go Barefoot syndrome.  But it's past time to
fix that.

I really don't care overmuch whether it's Chef, Puppet, Ansible, Salt,
or cdist.  (Just no cfengine.)  As with most things, the perfect is the
enemy of the good-enough.  cdist, despite its zero presence in Big Dumb
Enterprise Computing, might be just the thing because it's small and
simple, even more than Ansible.

Tiny problem:  The only ones I'm really familiar with are Puppet and
Chef (and cfengine).  So, the path of least resistance is to use the
familiar and what one is likely to work with for a living, but if I
swallowed that way of thinking, I'd be running CentOS.  Or Windows
Server.  Or Solaris.  ;->





> qemu-kvm - done lots 'o that on Debian - and for years.  Highly
> recommended for VM infrastruture.  One *might* want to *also*
> install virtualbox ... 

No.  No.  No.  A thousand times no.

This is a security-focussed production server, with careful minimalism.
VirtualBox would be a clueless choice, totally defeating my objectives.

For the same reason, no VMware, no Xen (OK for its time, but KVM/qemu
has fewer problems, and no other pinheaded software-selection errors.


[VM neepery:]

> Once you've set that up, I can make some general recommendations
> (e.g. [virtual] network bits, VM format, etc.) - even have a very
> handy TEMPLATE file I use for quick and convenient creation of
> virtual machines - copy it, tweak some common settings, execute
> it, and VM is created up and running.

Yeah, I'll definitely want that.

> DNS nameservers ... RFC-1918 - don't need two dedicated hosts

No.  Definitely not.

It's not IP shortage.  I have a /29, so five public IPs, but wasting any
is dumb, and part of the aim is to keep attack surface microscopic.

> can put both on same interface or [v]LAN - no need for separate host;

We'll have to talk more about this, as I have a feeling it'd be a lot
quicker and clear than as part of a mailing list thread.

FWIW, I'm not _used_ to doing VLAN tagging in Linux, but doubtless will
have to run with the concept since I'll be relying on KVM/qemu.
Something new in my life, yay.


> Unbound on RFC-1918 IP
> neither listening on wildcard addresses - bind to specific IP(s)

Maybe.

Best practices, done thoroughly, dictate not even having recursive DNS
on the same host as does the public-facing stuff at all.  But so far,
every time I try to plan for that in my existing house networks (note
plural), I get a headache and don't come up with something that meets
all my objectives.  And then I run out of time, because, e.g., the other
members of my house just arrived home from shopping, except they've left
everything out in the car, and will I please drop everything I'm doing
and go out and get it for them?  Thanks, Rick.

> I don't know about NSD, but if it's capable of doing so, and
> you want/need to have it also do recursive from selected
> Internet addressable IPs/blocks, if it can do it, it may be
> possible to have it selectively forward stuff it can't itself
> answer and only do so for queries from specified source IPs ...

As a matter of deliberate design, NSD does only authoritative DNS and
not recursive.  As a matter of deliberate design, Unbound does only
recursive DNS and not authoritative (except a funny little optional
feature called 'stub zones' that I do not intend to use).

Again, best practices is to completely separate authoritative from
recursive.  And I'm not fooling around this time; I intend to do just
that.

Compromises exist such as running both daemons on a single host, but
with dnsproxy answering port 53 and sending queries to which of the two
daemons is appropriate.  That's clever, but also a bit stupid, in the
sense of partially defeating the entire objective of separating the two
functions, and also adding avoidable complexity to system design.  (Part
of my design philosophy is that the simplest possible implementation is
the one you're more likely to understand, least likely to fail in a
bizarre and nondeterministic way, and most likely to have good security
and reliability prospects.  So, when I hear a proposed solution like 'I
know!  Run three daemons on one host to provide two DNS services!', I
think somebody is not clear on what simplicity means.


> Did you know (I'm sure I've mentioned it a time or two ... perhaps else-list)
> one can "lock" modules in - so once kernel is up, running, and
> "locked", after that, no modules can be added or removed.

But it's simpler to just have no capability to load modules at all,
because that ability has been deliberately omitted from the kernel one
compiles.  See?  Simplicity.  I'm not fooling around when I say I'm
going for that.

For over a quarter-century, I've gotten looks of horror when I mention
the possibility of, once again, making a custom kerenel that is locally
compiled to include only and exactly what is needed for the specific
hardware and for specifically desired host functionality.  'But why
would you do that, Rick?  Isn't it easier to be able to decide one day
to run NFS client and be able to just modprobe the driver?'  Oh, fuck
off, basically.  No -- for the host OS's kernel -- I am not going to
consider it to be in-scope to have dynamic loading and unloading of
modules of any kind.  Needed drivers will be compiled inline, period.
Again, not screwing around.  The (base) host OS in particular will be
one where anything that's not part of the initial design scope will be
just not there, to the extent reasonably possible, and any unused
functionality that's too much trouble to omit entirely from the system
build will be deliberately disabled with superuser finality.

Most people want to have extra stuff sitting around that they _might_
want to use some day, because basically most Linux users and even many
sysadmins are gadget freaks.  As to the host OS in particular, I am
absolutely not.  I'm going to be hardass in the opposite direction.
If I'm not reasonably certain I'll be actually needing something, it
will be tossed and preferably not even present as disabled code.


With luck, the above will at least convey the flavour of where I'm
aiming.