[sf-lug] wish? to sole? operator/hoster of majority of ... (was: Re: spam vs. anti-spam(2))

[Michael Paoli]

(okay, so maybe the subject/topic meanders some fair bit ... but still mostly
regarding the SF-LUG list and (mostly) related(ish))

> From: "Rick Moen" <rick at linuxmafia.com>
> Subject: Re: [sf-lug] wish? to sole? operator/hoster of majority of  
> ... (was: Re:      spam vs. anti-spam(2))
> Date: Sun, 9 May 2021 19:13:26 -0700

> Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
>
>> And, relatively early along the way, the SF-LUG list got hosted on
>> linuxmafia.com.  As I understand it, that was to be "temporary"
>> solution/fix/resource, as somehow Jim was dealing with list somewhere
>> else
>> https://web.archive.org/web/20040813161324id_/http://virgilsoftware.com/mailman/listinfo/sf-lug
>> https://web.archive.org/web/20051029224239id_/http://virgilsoftware.com:80/mailman/listinfo/sf-lug
>> and had it failed in some manner and without backups - at least I'm
>> presuming, that Jim couldn't get it going again or restore that data.
>> https://web.archive.org/web/20051215145247id_/http://virgilsoftware.com:80/mailman/listinfo/sf-lug
>> But, uhm, yeah, earlier, apparently nobody had backed up the list
>> before it was on linuxmafia.com
>> http://linuxmafia.com/pipermail/sf-lug/2019q2/014130.html
>
> The weird thing was that Jim completely ignored me, over and over, for
> years, whenever I politely inquired about what had happened -- even
> though I had stepped in and saved the group's ass, which frankly strikes

Yeah, over the years, I don't now that any of us have ever heard from
Jim, or anyone else, "exactly" what happened to the earlier SF-LUG
list.  It would seem obviously things weren't backed up - or at least
in manner that Jim had or had access to.  As I would guestimate if he
had that or access to such backups, he would've at least offered up
that data for list to be restored - to wherever - or might've even
asked for assistance on access to backups or data that otherwise
effectively included that ... don't think I've ever heard a peep - even
historically, regarding such.  But I think too that predates my being
involved with SF-LUG - so maybe something happened ... even off-list or
conversation or whatever, that I never heard about.  Who knows.  Also
no idea if the earlier list was hosted by some company or organization,
or if somebody just volunteered it - and no idea what fault/issue
caused it to go non-functional, at least for the sf-lug list thereupon.
Maybe somebody knows or remembers, or would want to research and
report on, what virgilsoftware.com (and Mailman thereupon) was back
around the time frame that the SF-LUG list went bye-bye on there
(2005Q4) for whatever reason(s)/cause(s).

> me as pretty seriously ungrateful.  And then _lately_ he claimed that

Yeah ... non-response would be not good ... maybe embarrassed? ... or
dear knows what (and/or what else) may have been going on at that time.

> there _never had been_ such a prior incarnation of the mailing list --
> until I responded quoting him from 2005 on the subject about there
> having been that earlier incarnation, its unexplained-by-him sudden
> catastrophic vanishing and the seeming total absence of backups, and
> consequent need for new hosting.

Yep, ... ain't nobody's memory perfect (though for some rare few, it's
damned impressive).  But hey, The Internet Archive does relatively
objectively remember at least some of it.  But alas, archives of the
much older (before linuxmafia.com) not public ... so mostly "remembers"
existence ... and non-existence, and some whens.

> Having corrected the record (again), I (again) asked what happened back
> in 2005 -- and now Jim is back to steadfastly ignoring the question, as
> before.

Yes, we've seen the posts.  :-)  Questions remain.  Don't know if we'll
ever get answers - seems the more time that passes, the less probable
that is - at least to a statistical correlation probability level,
anyway.

>> Oh, and, also moved BALUG's list hosting - which had been on
>> DreamHost.com - not only was there the expense there ... but they sucked
>> at it, screwing up BALUG's list on multiple occasions, losing data,
>> being unable to restore it, not proving a means for us to back it up, in
>> addition to numerous outages, etc. ... so, yeah, BALUG got the heck off
>> DreamHost.com.
>
> Context for that:  In a nutshell, Dreamhost is a specialty _WordPress_
> hosting firm -- one of a number who compete in that space (Bluehost,
> wpengine, etc..  Such firms are highly focusssed on the ongoing
> requirements of doing software maintenance for the
> security-problem-prone WordPress PHP codebase and its large extended
> market of extensions and themes.  Such firms are _also_ notorious for
> sucking at other basic ISP functions, such as at doing SMTP e-mail and
> hosting mailing lists.

> Seeking out a specialty WordPress hosting outfit to provide mailing list
> hosting isn't _quite_ as bad a decision as going to Fort Mason's Greens
> Restaurant ("fine vegetarian cuisine") for a steak dinner, but IMO
> nearly so.  It's a pretty good recipe for becoming an unhappy customer,
> so I'm extremely not surprised that the utterly predictable happened.

Well, for bit more background on BALUG and DreamHost (not SF-LUG
relevant, skip this (long) paragraph if you're not interested in that
bit), BALUG had for quite some time prior to that been hosted on
DreamHost.com - wasn't my decision at all - someone else did that - not
sure where things were hosted earlier than that (at least I certainly
don't recall off-the-top-of-my-head, and also predates me having any
BALUG involvement other than having attended, probably joined list, and
did my first BALUG (and first LUG) presentation at BALUG on ...
2005-05-17:
https://lists.balug.org/pipermail/balug-announce/2005-May/000032.html
).  Anyway, prior to about 2013-07-13, DreamHost.com services had been
relatively fine ... sure, some glitches/issues, but nothing catastrophic
or from which they(/"we") couldn't well recover from.  That all changed
in 2013, and continued to be a repeated hot mess until we fully
extricated ourselves from DreamHost.com.  So, e.g., exhibit A:
$ cat */archive_date_ranges
2005-03-18--2013-05-24,2014-01-11--2015-01-31,2015-05-01--2015-11-30,2016-02-18--
2001-06-15--2013-07-12,2013-11-18--2014-09-30,2014-11-14--2015-01-31,2015-04-20--2015-11-30--2015-12-15,2016-02-15--
2001-06-15--2013-07-13,2013-11-09--2014-10-19,2014-10-22--2015-01-31,2015-04-06--2015-12-05,2016-01-23--
$
See those dates - dates of archives - 3 lists, 3 sets of dates.
And the dates and ranges -- (alternative to /) per ISO - showing a range
of dates.  Most notably see the commas (,)'s in there.  Those are major
DreamHost.com screw-ups - big time.  Those are places where they lost
our archives and could not restore them - gone.  Not the way to run a
commercial service that sells listserver services as a service - not at
all - not even close.  Totally unacceptable.  No idea what
DreamHost.com's problems were, maybe lack of sufficiently clueful and
skilled staff, but I'd have to push that blame to management/ownership.
Basically their service went to sh*t.  Not only failing very badly, but
repeatedly so.  They also made it bloody difficult to backup - no direct
access to complete archives in raw mbox format, not even as a listadmin
or through their console.  Had to be specifically requested as a
one-off, to support, and only by authorized person(s) having quite full
access to the DreamHost.com account.  What a crud operation it was (or
had very much become).  So, BALUG, we carefully extricated ourselves
from there.  Likewise since they wouldn't at all gives us access to the
configuration files, etc., to migrate things as painlessly as feasible
for the many (well over 800 I think even then) list members,
specifically wrote web scraping software to get absolutely as much
configuration as possible - notably all the user's settings/preferences,
except for their list password.  So, with a one-off special request to
get the raw mbox files - which was like pulling teeth to get it from
them - got those ... and with the web scraping software I wrote, got
everything else except for users' list passwords.  And with that, we
migrated the heck off of DreamHost.com.
https://www.wiki.balug.org/wiki/doku.php?id=balug:mail_and_lists
Oh, some additional details I'd forgotten ... yeah, only /the/ primary
account holder of the DreamHost.com account could request the mbox
file copies - DreamHost.com had zero provisions for making it possible
for anyone else to even request those, and requesting them was only
available by specifically opening a support ticket for such.
Also, not only web scraping - I believe I also wrote web software to
essentially do the inverse to load it to the target location - I must've
either done that, or somehow worked out way to get that loaded into
Mailman's Python 2.x .pkl (pickle) format data files ... and with me
not coding in Python at that time.  So ...
Friends don't let friends use DreamHost.com.  Partimus.org, last I
checked still uses DreamHost.com.  Partimus.org folks continue to
grumble about DreamHost.com, I continue to warn Partimus.org folks of
the impending risks/dangers of remaining there ... but thus far they've
still remained there.  Well ... not my circus, not my monkeys.  ;-)
Also ... DreamHost.com ... WordPress?  Well, DreamHost.com did all kinds
of stuff ... including offering dedicated hardware servers - BALUG even
had that once-upon-a-time (but later switched to hosted shared services
there - again I had noting to do with that change or choice).  But
looking back around the time things turned to crud with DreamHost.com
https://web.archive.org/web/20130524055418/http://www.dreamhost.com/
though it states "WordPress-optimized" and in at least some locations
they list WordPress first, it's not exactly super highly promoted that
that's their specialty, or anything close to stating or implying that
that's all they do ... as they do (or at least did) quite a bit more -
and promoted and very much sold those services too.  But yeah,
interesting to know that they did first and foremost list WordPress -
even if they don't exactly have it written up to jump out at folks ...
don't think I was aware of that, or perhaps I'd subsequently forgotten
that detail (and don't think BALUG ever did WordPress on DreamHost -
and maybe/probably never even did WordPress at all - so may never have
jumped to or been particularly called out to my attention).  Anyway,
enough about DreamHost.com - glad that remains well behind BALUG and
not under or in front of BALUG (or really any LUGs where I deal with
their hosting or email or lists - at least beyond being subscribed).

>> But, e.g. when that laptop is going to go out with me or otherwise not
>> be available for such hosting at home ... fire up vicki and live
>> migrate the VM to there (and later migrate back to kill the noise and
>> save power).
>
> I just wanted to take a moment to admire this cool whole-system
> live-migration capability.

And too, must also give credit to virsh and friends.  One of the cool
things there that makes it possible, is the --copy-storage-all option.
Much commercial VM software doesn't even have equivalent capability.
With --copy-storage-all one can live migrate a VM where source and
target hosts of the VM share no storage in common!  For most VM
software, live migration requires the hosts have storage in common, e.g.
Storage Area Network (SAN), NFS, clustered filesystem, etc. - some
common storage where both hosts have access to the same shared storage.
Well, with --copy-storage-all that's completely unnecessary.  With that
option virsh takes care of doing live migration of the storage - without
it even being in common between the two hosts.  As I understand it, to
pull that cool "trick", it uses Linux's network block device capability.
It essentially, on-the-fly, switches the storage from local, to network
block, mirrors the storage on that between the two hosts, then once sync
is achieved on that mirroring can continue with any remaining live
migration steps.  And after the VM itself has completed live migration,
to wrap things up it breaks that network block device RAID-1 mirrored
storage and returns it to local storage - but now on the target host.
Pretty dang slick.  And very practical.  No need for some separate
storage architecture or dependencies.  So well suited where that may not
exist, or might not exist when some particular live migration is needed
for whatever purposes or under whatever circumstances.  Got network
connectivity between the machines ... potentially even indirect ... good
to go.  (I do it over ssh ... so ... anything one can transport over ssh,
including indirect ...).

>> And, as for the SF-LUG list ... theoretically the linuxmafia.com
>> is just "temporary" ... 'till Jim (and/or SF-LUG) gets its act
>> together to do its own hosting/coverage of that somewhere ...
>> not that Rick is exactly chasing SF-LUG off of there, but does
>> sometimes get rather annoyed at the "demands" made/asked of him
>> for the services he graciously provides to SF-LUG of the list
>> hosting....
>
> To be clear, this hasn't been a problem _lately_ -- but, yes, my
> pleading with the SF-LUG principals to please pick up the offered
> periodic backup of mbox file + roster for -years- and getting totally
> ignored, and _then_ Jim acting like I'm his personal servant for
> delivering a backup to him while my server was down because of a
> hardware failure, _that_ definitely was pretty galling.

Yes ... good to (at least currently) not be a problem ... and hopefully
remains the case in future too!  :-)

> Jim separately and additionally maligned in public my Internet operation's
> ethics, just because he had failed to understand that volunteering
> to be a Mailman admin meant he'd be getting notices of held spam in the
> admin queue.  That _also_ was objectively infuriating.

Ugh, yeah, not cool.  Blaming the volunteers / what's volunteered, is
generally not the answer.  Also good to not bite the hand that feeds one
(and thanks for also helping feed CABAL folks - I'm sure it was most
yummy again, although I didn't make it in-person to this past Saturday).

> Nor have I ever gotten even after-the-fact apologies for any of this
> stuff, just lame attempts to talk around that and justify it.

Sounds rather like excuse making ... or more like avoidance.  :-/

> The time-honoured way of alienating volunteers is to ignore their polite
> requests, then try to treat them like servants when suddenly _you_
> have a problem.  SF-LUG has has provided textbook examples.  And then
> kicking my system's reputation for running a clean antispam operation in
> the teeth was adding injury to insult.

Yeah as I say, volunteers/volunteered, hand, bite, etc. - don't do that.

> But I haven't kick y'all's asses off.  I've merely said those things
> were rude, annoying, and unacceptable -- and that "We're all just
> end-users who don't know what we're doing" doesn't excuse it.

Yep, "nobody" (well, most of the time, and some exceptions here 'n
there) stepping up, yeah, that's not good, and isn't really any
excuse for that.  Somebody(s) really needs be (sufficiently)
responsible and step up.  Well, good that, e.g., at least some folk(s)
besides Jim (and Rick) have been reasonably covering listadmin duties
for the SF-LUG list.

>> And thanks to Bobbie for managing an off-list distribution list to
>> keep the "list" happening more-or-less via email until list proper was
>> back on-line.  And thanks to Bobbie and Daniel for well saving those
>> emails and providing 'em to Rick, so Rick then subsequently merged
>> those off-list "list" emails into SF-LUG's list archive.
>
> I appreciate Bobbie and Daniel doing that, too.
>
> Just to make the point once again, when I made the offer of merging in
> the offlist discussion, what I said was that I'd be willing to do that
> if the mail were provided in mbox format.
>
> How many of those couple of dozen messages were furnished to me in mbox
> format as very clearly specified?  Exactly zero.  _None._
>
> Basically, I was handed a crazy-quilt of weird message formats, not a
> single one of them being mbox.  It was basically, "Oh, here's a bunch of
> stuff in whatever format I happened to have.  Good luck figuring out how
> to transform it into what you need."
>
> I figured out using sed and awk, over a few days of poking at the
> material and then verifying that the result of my scripting was valid
> mbox-type mail, that I has _transformed_ all of that near-rubbish into
> something I could work with.  But that was a whole lot of hours that I
> would rather have used to work on my _own_ tasks, rather than being
> drafted implicitly into doing a bunch of work I hadn't volunteered for,
> just to do the favour I _had_ volunteered for.
>
> And before somebody says "Oh, but we had _no idea_ what an mbox file is",
> guys:  (1) You bloody well could have _asked_, or (2) Web-searching the
> term "mbox" brings up a correct answer on the first hit.  (mbox is,
> among other things, the native mail-storage format of Mozilla
> Thunderbird and many other mail programs, and has been used as a primary
> mail storage format in Unixes since 1974.)

Hmmm... maybe you could'a asked for a bit more assistance on that.
E.g. "I requested" ... "but what I got was" ... "you can grab copies of
what I got here" ... "on my website.  If someone(s) wants to put it into
the requested (and in fact required) format, then" ...
Anyway, I might'a taken a crack at that ... and/or maybe other(s)
would'a potentially covered that.  Sorry you (also) got stuck with that.

Anyway, thanks that you did in fact do the work on that - in case others
haven't already quite said so.

>> Other than Bobbie doing list work-around while it was down, and Bobbie
>> and Daniel later providing those email archives to Rick, I don't think
>> anyone else lifted a finger (though there may have been theoretical
>> discussions of finger lifting).
>
> Not even that.

Dang ... that sucks.  (well, other than the separately mentioned work
Rick and I also did to fix what needed to be fixed to get the list
back on-line again as it was before the failure(s) that caused it (and
the host) to go (hard) down).

>> Anyway, as I do also host BALUG lists on the BALUG VM and using
>> Mailman 2.x software - about the same as linuxmafia.com, thought
>> I might offer to have that moved over.  That would improve the DKIM
>> situation, as linuxmafia.com has older version that doesn't play
>> nice with DKIM (well, DKIM really doesn't play nice period with
>> lists and such) [...].
>
> Just to be (once again) clear about the scope of the real-world problem:
> The problem gets triggered by any posting from a subscriber whose mail
> domain has DMARC policy p=reject or p=quarantine.  The only ones of
> those I know are Oath, Inc.'s two domains yahoo.com and aol.com (which
> have p=reject), and Apple's domain me.com (which has p=quarantine).
>
> Postings from @yahoo.com and @aol.com subscribers arrive after
> transiting mailman at _other_ subscribers' domains that check and
> enforce DMARC/DKIM (such as GMail) with, effectively, a request that the
> receiving ISP please refuse the subscriber's copy of the @yahoo.com or
> @aol.com person's posting (because the DKIM signature no longer
> validates after transiting Mailman).  When that refusal happens,
> linuxmafia.com's SMTP process correctly reports a "bounce" back to
> Mailman, and the receiving subscriber's Mailman bounce score gets
> incremented.  (The person also doesn't receive the posting, of course.)
> When this happens frequently enough, subscribers' mail delivery gets
> disabled for excessive bouncing, and eventually they get unsubscribed.
>
> So, basically anyone who posts from a @yahoo.com or @aol.com posting
> address shoots _other_ people in the foot, and also ensures that
> affected subscribers will never see those postings.
>
> Apple's domain me.com is an odd case.  Having p=quarantine set in the
> me.com DMARC policy, by _itself_, is merely a request that
> DMARC-compliant receiving domains spambox the suspect mail (quarantine
> it).  Normally that ould not result in receiving members at
> DMARC-observing domains getting incremented bounce scores, but it turns
> out that, for some reason I don't fully understand, GMail (at least)
> was 550 _refusing_ a me.com poster's mail sent out via Mailman (rather
> than just quietly spamboxing it).
>
> Anyway, to sum up, historically big providers _other) than Oath, Inc's
> two (them having been the authors of DKIM and DMARC) have in my
> experience been smart enough to _avoid_ declaring aggressive DMARC
> policies -- probably in part because of DKIM's hapless hostility towards
> (and damage) to mailing lists.  (There have been a few smaller domains
> that made that error, though.)
>
> Until that weird example of damage from a me.com subscriber posting to
> the Skeptic mailing list arose, I'd never seen elevated bounce scores
> from a domain with p=quarantine.
>
>
> By the way, there's at least one sf-lug at linuxmafia.com stalwart who
> posts from an @yahoo.com address.  John, your choice of mail provider's
> entirely your affair, but you know you elevate many other subscribers'
> bounce scores, and also you know they won't see your postings, right?

DKIM ... I don't know that blame ought go to users that use such
ISPs or email service providers that inflict such DKIM problems upon
lists, etc. - but sure, if they're aware of it and continue to use it
for such, etc., sure.  But some folks may be quite attached to such
email addresses/services - may have even had 'em long before DKIM even
existed.

Anyway, yeah, DKIM - problem - and more so for the somewhat older
Mailman 2.x series software on linuxmafia.com.  And ... what's to be
done about it?  Well, so long as the list remains there, and there are
members/subscribers with such problematic email senders, and they post
to the list ... well, that's a problem ... but not really my place to
say how to best handle that where it's presently hosted.  I'm sure Rick
has no shortage of opinions and such regarding what ought be done.  :-)
Oh, and how many ... since I also have roster access (notably also
regularly (daily!) back it up ... so, let's see ...:
$ hostname --fqdn
balug-sf-lug-v2.balug.org
$ pwd
/home/sflug/sf-lug_roster
$ id && ls -ld --full-time sf-lug_roster
uid=29774(sflug) gid=29774(sflug) groups=29774(sflug)
-r-------- 1 sflug sflug 8150 2021-05-08 14:04:26.000000000 +0000  
sf-lug_roster
// And, FYI, with version control 'n all, the mtime and ctime timestamps
// on the above remain the same if there are no changes to the data.
// In the below, we can see evidence of the download and comparison
// having been done - as that changes the mtime and ctime of the
// directory, had the data differed, new file would've been checked into
// version control, but with identical matchng data, the newer file
// having identical data is simply removed and older left unchanged.
$ stat -c '%y %z' .
2021-05-09 14:30:10.967899612 +0000 2021-05-09 14:30:10.967899612 +0000
$ wc -l *roster
258 sf-lug_roster
$ sed -e 's/^.*<\([^>]*\)>.*$/\1/;s/^.*@//' *roster | sort | uniq -c |  
sort -bnr | wc -l
84
$
// Wow ... rather high diversity of domains relative to number of
// members/subscribers ... so, won't show 'em all ...
$ sed -e 's/^.*<\([^>]*\)>.*$/\1/;s/^.*@//' *roster | sort | uniq -c |  
sort -bnr | sed -ne '/^ *1 /q;p'
     148 gmail.com
      17 yahoo.com
       7 hotmail.com
       4 sbcglobal.net
       2 protonmail.com
       2 open-source-staffing.com
$
// Those are all the domains with 2 or more subscribed addresses.
// And known/probable problematic ones? ...
$ sed -e 's/^.*<\([^>]*\)>.*$/\1/;s/^.*@//' *roster | sort | uniq -c |  
sort -bnr | fgrep -i -e yahoo -e aol -e me.com | fgrep -i -v  
dslextreme.com
      17 yahoo.com
       1 yahoo.co.in
$
So ... likely 18 - of 258 ... that's fair percentage ...
$ echo '18/258*100' | bc -l
6.97674418604651162700
$
So, right around 7% - not exactly a trivial percentage.