[conspire] Don't you just hate it when ... you're right! <sigh>

[Michael Paoli]

So, $work ... bit 'o back story.
$work was seriously interested in spending a non-trivial chunk of
change, oh, like about half a million USD, on some commercial software
"security" solution product ... to implement something really not
all that incredibly complex.  I mean sure, the software had all kinds of
bells and whistles too, and additional features, oooh, and even support
probably included in that price (or maybe for the first year?).
But none that were required, nor even were of particular interest
(other than maybe support?).

But essentially all they wanted it for was for something relatively
simple.  So, what did they want to be able to do?
About this.
Let's say you have some set(s) of secrets you want to securely store.
Okay, you can encrypt in storage (on drives/disks).  Okay, well then
you still have to manage the keys/passphrases/secrets (let's just call
'em secrets) to decrypt the storage.  And maybe you want/need that to
happen relatively easily and quickly and fast automagically - e.g.
at boot - so that means that secret is readily accessible to the host -
e.g. like it's on some USB device connected to the host.  Uh huh.
So, if someone comes in and steals the physical host and that attached
USB device, you've still got a problem, 'cause they can now decrypt all
that data - including all on the drive(s).  Okay, so, how do you fix
that?  Ah, securely split it among 2 (or more) hosts, such that the full
contents (and attached USB device) would leak absolutely no data about
the actual stored secrets (could decrypt the drives, but ...).  And how
to securely do that?  Easy - One Time Pad (OTP)
https://en.wikipedia.org/wiki/One-time_pad
... or possibly anything else deemed sufficiently secure,
but for example we'll use OTP - and presume the actual secret bits
aren't too huge (e.g. password and passphrases and secret keys and
the like).
So, have OTP of length not less than the secrets to be protected.
eXclusive OR (XOR)
https://en.wikipedia.org/wiki/Exclusive_or
the secrets with the OTP.
One host stores (encrypted on drive of course, but whatever) the OTP,
the other, the results of the secrets XORed with the OTP.
If either host is stolen (along with the USB to decrypt drives),
the data is still fully safe, as it's still (secure OTP) each provably
secure - as long as they don't get both, all is good.
What if one wants higher levels of security?  E.g. to require adversary
to steal 3 or more hosts to be able to compromise the secrets?
Repeat the process as needed - take the encrypted secrets on either
host, have another independent OTP (never reuse for
encrypting/decrypting other data), XOR it with the new OTP, and likewise
split - now on 3 hosts, and adversary would need all 3 to get to the
secrets.  One can repeat this as needed.
What about redundancy?  Don't want to lose all access to all the
secrets if one single host goes goes up in flames or whatever.
Replicate - more sets, OTPs, so one can do an N of M - M hosts,
need any N of M to access secrets, but less than N, and one only has
provably secure encrypted data - essentially nothing but independent
random (OTP & XOR) noise - have to combine N to decrypt.

There are more steps for managing all that, e.g. having means for,
e.g. Alice and Bob to be able to get access to the secrets, or just
some of them.  Likewise require, e.g. both for access, or N of M
authorized users for access, etc.

And more bits on the hosts for security ... at some points some of that
needs come together in RAM ... so, physical security still very
important, be sure secrets and related are locked in RAM, so they're
never swapped/paged out ... probably good
Defense in depth
https://en.wikipedia.org/wiki/Defence_in_depth
precaution to also completely disallow and not have any swap on any of
these hosts.  And also, indirection(s) ... e.g. a master key protects
the secrets, and N of M authorized users ... they never ever get access
to the master key - whatever they use and combine to gain access - it
just gets them access to the secret(s) they're authorized to access.
E.g. their passwords or the like just unlock or create some
(temporary-ish) sub-key that in turn unlocks the master key, and then
the authentication and authorization and such, only gives them access to
the secret(s) they're allowed to access (or to not access, but to deploy
or whatever).

Okay, not exactly rocket science.  Sure, need pay dang good attention to
some details to do it right and not screw up, but not absurdly hard.
OTP & XOR, possibly use other encryption algorithm(s) deemed
sufficiently secure, ... that's mostly it - along with physical
security, on-drive encryption, etc.

And ... what did I do some many months back at/for $work?
Proof-of-Concept (PoC).  I'm like hey, half a million bucks?
Are you friggin' kidding me?  And for only that key bit that you
care about?  I can give you a working PoC in about half a day's
time ... and so I did.
And of course thus far that's gone nowhere (I probably didn't spend
enough $$ on C-level exec lunches? ... or dear knows what).

Meanwhile, that "security" product ...
https://nvd.nist.gov/vuln/detail/CVE-2020-16250
(NIST) Base Score: 9.8 CRITICAL
So, yeah, 0 to 10 scale, 9 or above is CRITICAL - 10 as bad as it
gets.  They got a 9.8.  Definitely not where one wants to be.

"Everything should be made as simple as possible, but no simpler."
- A. Einstein

Take something that's that - highly important for security.
Add tons 'o bells and whistles and features and capabilities and
bloatware and stuff atop it.  What could possibly go wrong?

Oh, ... like 9.8 out of 10 bad wrong, that's what.
We see this far too commonly.
Sometimes I really hate being right.

I mean heck, we've never seen, oh, some bloat a highly
commonly used shell and then have a major security ...
oh yeah, GNU bash did that.
https://en.wikipedia.org/wiki/Shellshock_(software_bug)
Many many other examples, but that's just one more typical
one that jumps to mind.
Egad, systemd is another ... though hopefully by now most or
all of the major security booboos have been squeezed out of
it ... and won't get reintroduced, but ... egad, that friggin'
thing is pretty dang huge, and in a very security critical position.
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd

HashiCorp Vault ... (NIST) 9.8 out of 10 CRITICAL bad.
Yep, sometimes I really hate being right.