[conspire] Selected bits of SF-LUG (list) history, Mailman, linuxmafia.com, BALUG, etc.

[Michael Paoli]

> Date: Fri, 22 Jun 2018 14:18:56 -0700
> From: Rick Moen <rick at linuxmafia.com>
> To: conspire at linuxmafia.com
> Message-ID: <20180622211856.GA28236 at linuxmafia.com>
>
> ensued, which might be of interest here and also bears on topics
> discussed during this past week's BALUG meeting.
>
> ----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
>
> Date: Fri, 22 Jun 2018 14:14:34 -0700
> From: Rick Moen <rick at linuxmafia.com>
> To: [a valued friend on OSI's mailing lists]
> Subject: Re: [License-review] Fwd: [Non-DoD Source] Resolution on NOSA 2.0
> Organization: If you lived here, you'd be $HOME already.
>
> I wish to also make a point in passing about the _ease_ of arranging
> backup and failover of relatively simple, commodity Internet services
> like SMTP & mailing lists based on the GNU Mailman MLM:  Here in the Bay
> Area, I for many years hosted (and still host) on linuxmafia.com a
> Mailman list for a San Francisco LUG called SF-LUG.  During those many
> years, I repeatedly pleaded with the leadership of that group to work
> with me to set up periodic, automated backups of the membership roster
> and cumulative mbox -- with no result except avoidance and excuses.

Yes, oft posted/reminded.  The hosting software (Mailman), is among
other things, configurable to allow the entire raw mbox format file (the
raw total collection of all postings to the list) to be accessible in
manner quite similar to the more general archive of list postings ... so
generally public ... or possibly restricted to just list members?
("private").  In any case, the sf-lug list graciously hosted by Rick on
on linuxmafia.com is so accessible, and Rick has often made the point
that it's there, available to be backed up in its entirety, by anyone
that would so much as care to bother (my paraphrasing, not necessarily
an exact quote).  So ... linuxmafia.com lists ...
http://www.linuxmafia.com/
mailing lists --> http://linuxmafia.com/mailman/listinfo/
... archive ...
all but two "listed" (I think it's "advertised" in Mailman's
configuration terminology) lists there are publicly archived (two appear
to be members only access - password protected).  And of the public ones
... the main archive pages don't provide URL for downloading entire raw
mbox format archive file - but it's rather well known, and has been oft
mentioned (certainly at least for the SF-LUG list).  So, let's see ...
$ (for l in conspire dvlug friday-follies sf-lug test
do echo "$l" $(
curl -s -I http://linuxmafia.com/pipermail/"$l".mbox/"$l".mbox |
sed -ne '/^HTTP/{p;/200/!q;};/^Content-Length:/p' | tr -d '\015'); done)
conspire HTTP/1.1 200 OK Content-Length: 46875787
dvlug HTTP/1.1 200 OK Content-Length: 3139113
friday-follies HTTP/1.1 404 Not Found
sf-lug HTTP/1.1 200 OK Content-Length: 70806214
test HTTP/1.1 200 OK Content-Length: 22522
$
And so, we can see, that for the 5 lists publicly archived there, 4 of
the 5 have their full raw mbox archive files also publicly available
(and does include both the conspire and sf-lug lists).

> A few years ago, I had a couple of months of downtime because of an
> improbable motherboard failure right in the middle of a major system
> upgrade.  This happened right immediately following my losing my job, so
> I was a bit demoralised, and was then barraged by rather clueless
> complaints from SF-LUG insiders, e.g., suddenly demanding copies of my
> backup data but being unwilling to travel 30 miles to my house to get
> it.  (Apparently, I was supposed to drive to _them_, or something.)  The
> more the SF-LUG people barraged me with complaints and totally useless
> suggestions, the more I put off rebuilding my server, which actually
> wasn't that much work.  I figure I delayed about two months longer
> before bothering to fix the situation because of the annoying
> commmunication than I otherwise would have, as suddenly I was rather
> enjoying the vacation from running an Internet server giving
> free-of-charge services to ingrates.

Ah, nice to have vacation(s) ... sometimes even if, uh, "unplanned".

> Tellingly, the reason I offered hosting to SF-LUG's mailing list in the
> first place is that the main guy _had_ created a Mailman list, ran it
> for a brief while, and had some sort of hardware failure that caused the
> total loss of everything.  Nobody had even so much as bothered to
> occasionally copy the cumulative mbox off to a USB flash drive, which
> would have been dead-simple and was totally flippin' obvious.  The guy
> was devastated and demoralised by the experience -- but, as became
> apparent later, learned nothing.  I wished the group well and so
> gladly offered it a mailing list home to replace the lost one -- and

Wow ... I didn't know that earlier bit of history ... notably before
sf-lug list was first hosted on linuxmafia.com.  Surprised (or perhaps
I'd forgotten?) but ... yeah, not (that) surprised.  I mostly got more
(at least somewhat) involved with SF-LUG sometime >=2003, and maybe more
like in the 2005-2008 range, when I started assisting with some
(non-list) stuff that SF-LUG had hosted in a colo - yeah, looks like
that was <~= 2007, as looks like it was around 2007-08 when SF-LUG got
the 1U Super Micro:
http://linuxmafia.com/pipermail/sf-lug/2007q3/001883.html
et. seq. (and which subsequently became the "vicki" host (hostname),
which I helped out on quite a bit) ... before "vicki", I'd helped SF-LUG
with stuff on some "tower" machine that Jim Stockford had in colo, but
I'd guestimate that probably only went back <~=2 years prior to "vicki".
... I see my very first post to the sf-lug list was on 2006-07-22.
And "vicki" does still live - though SF-LUG no longer has anything in
any colo.  The vicki host is in my residence, and that host is mostly
only used when my personal laptop is not online at home - as vicki is
annoyingly loud, and relatively power hungry.  There is balug virtual
machine (VM) at my residence ... which also hosts SF-LUG's
non-list content.  When I'm to be out or my personal laptop down,
vicki gets powered up and that VM gets moved over onto vicki, but in the
more nominal case, that VM resides and runs on my personal laptop.

> immediately got ignored, for years and years, as I repeatedly implored
> them not to make _again_ the mistake that had destroyed them the first
> time.  Which rather tells you something, nei?

Oye, ... crickets ... we've oft heard the stories.  :-/

> When I _did_ get my linuxmafia.com server back online after about four
> months, a technical person _not_ among the somewhat useless people
> heretofore running SF-LUG stepped forward and worked out _really simple_
> means to make the membershop rosters periodically sent offsite.  (He
> also gave key help in recovering my server from its failed software
> upgrade, which helped motivate me to bother.)  For the rosters, it's a
> simple weekly cron job that e-mails the rosters to a list of people.
> For the mbox, it's (IIRC) an rsync cron job.  Very simple, highly
> reliable, _and_ provably that is totally sufficient to rehost the SF-LUG
> mailing list if a meteor lands on my server.

Yes, once linuxmafia.com was operational again, first orders of business
- at least for me - was doing some reasonable backup of the SF-LUG
critical list bits - notably raw mbox mail archive, and roster of
subscribers.  The mbox was already and had long been available to copy -
and Rick even had on-site backup copy(/ies) of sf-lug list data
(including at least raw mbox data and roster).  Rick also started weekly
emailing myself and Jim Stockford the sf-lug list roster.  One could of
course also grab the raw mbox archive via http, and could even grab
latter portion of it with continue capabilities (e.g. curl(1)'s -C
option).  But I wanted (at least for my purposes) somewhat better
solutions ... and also wanted to make any additional changes as easy as
feasible on Rick.  So ... roster - got Rick to also make that available
via - I think http if I recall correctly ... yes, http ... but not in
the clear - I provided PGP public key, so daily cron job runs, produces
roster, encrypts it using key, then makes it available via http.  And, I
think even a bit before that, I got Rick to also make the sf-lug raw
mbox available via public rsync.  And why rsync?  If ever anything is,
for whatever reason(s), added to or removed from, anywhere in the
archive, rsync will pick up those changes.  Rick also had public rsync
on linuxmafia.com, so negligible burden to offer up one more file on
that. On the receiving end, I set up programs and cron jobs
to gently fetch the data daily (overnightly), and also feed it into
version control (RCS).  So, I not only get the current/latest data, but
I also retain the historical, and do so in a relatively (notably space)
efficient manner.  And, that is how sf-lug list critical data (raw mbox
+ roster) gets off-site backups done.  Those are in turn, on balug
VM, which in turn gets its own backups done, including offsites and
rotations thereof.  I think Jim Stockford also still gets weekly
emailing of the sf-lug roster - I asked Rick to drop me from same, after
I had the daily pickup of the list going & verified, using gpg & http,
etc.

Oh, and having that raw mbox stuff available can also be quite handy.
E.g. I wanted to reply to and cite various bits scattered across
numerous posts in the list.  I could've pulled that stuff out via email
client - but hey, I figure that's all in that raw mbox of the list -
which is publicly accessible.  Well, I didn't really want/need the whole
thing ... but ... I didn't have to grab that much of it.  So, gee ...
how big is it?
$ curl -I http://linuxmafia.com/pipermail/conspire.mbox/conspire.mbox
HTTP/1.1 200 OK
Content-Length: 46821707
Much more than I want, and more than would be convenient for my or
Rick's bandwidth, but, okay, so how 'bout I get about the last 2MiB of
that:
$ echo '46821707-(2*1024*1024)' | bc -l
44724555
$ curl -s -C 44724555  
http://linuxmafia.com/pipermail/conspire.mbox/conspire.mbox >  
conspire.gt.44724555.mbox
$
And then I could (and did) relatively conveniently use that to pull out
the postings on the thread I wanted to reply to, and discard all but the
relevant bits I wanted to reply to (which were scattered across many
postings).

Oh, and for those that might be curious to read a bit more about the
(much earlier) getting linuxmafia.com back on-line and operational
again:
http://linuxmafia.com/pipermail/sf-lug/2015q1/010647.html
http://linuxmafia.com/pipermail/sf-lug/2015q1/010663.html
Though the above doesn't as much cover all the "noise" while
linuxmafia.com and the SF-LUG list were down.  I believe there are more
bits of that on the sf-lug list (off-list distribution mails during
outage later injected into the sf-lug archive) and I believe also the
BALUG-Talk list ... but alas, those aren't (yet/presently) on-line,
due to DreamHost.com repeated f*ckups with BALUG's lists.  (I have much
of the data, though in non-ideal form (not raw mbox) - hope to
eventually get around to adding it back to BALUG's archives).

Some other notes:
BALUG:
   I host the lists (https://lists.balug.org/), no more DreamHost.com
   f*ckups to mess with that.
   The raw mbox files are publicly accessible, so anyone can access them
   - e.g. back them up.  With that configuration option enabled, the
   version of Mailman on the balug VM also lists the URL for the raw mbox
   archive on the archive page.
linuxmafia.com:
   I'm guestimating the older Mailman software there doesn't
   automagically list the URL for the full raw mbox archive when the
   configuration option allowing access to such is enabled - but I'm just
   guessing.
SF-LUG:
   Yes, I backup the list roster and raw mbox data (as other than Rick's
   fine hosting and on-site backup(s), nobody else seemed to ever bother
   to do it).  I've also offered access to the roster data, but would do
   so over more secured means (mostly to make it a bit harder for spam
   bot harvesting, and the desire of many list members to not expose
   their list subscription status to non-list members).
BALUG & SF-LUG:
   I also have, or intend to have, the raw mbox files available via
   public rsync ... I know I put it on my todo list ... don't recall if
   it's done, or (probably) still pending.  In the case of SF-LUG, it
   would be non-primary source (as it, of course, first gets the data
   from linuxmafia.com).