[conspire] 1:2.1.29-1+deb10u5? Re: upgrade-in-place to Mailman 2.1.30 and want to test Mailman3?

[Michael Paoli]

So ... did quite a bit of investigating and testing,
most notably how might it be feasible to get linuxmafia.com
to mailman 1:2.1.29-1+deb10u5

First I tried relatively simple approach.
So ... testing on linuxmafia2 - another VM on same physical host,
notably for testing.
I grabbed fresh snapshot of the production linuxmafia.com VM
(alas, that could be easier, but was done in any case, perhaps
more on that later).  Applied those disk images to linuxmafa2,
so mostly identical VM hardware and starting with identical disk
images.  Then some changes to linuxmafia2, notably including move it to
NAT/SNAT subnet, add firewall rule to block outbound TCP 25 to all
except 127.0.0.0/8 and ::1, reconfigure host's network for DHCP (which
the NAT/SNAT subnet provides), and then once that's all set, lastly
enable the network interface (normally disabled at VM level, 'caue it's
so similar to prod as would be problematic - same data, same Ethernet
MAC addresses, etc.).

So, from that basis ... try to get to mailman 1:2.1.29-1+deb10u5
installed.  So, got the .deb fine.  Inspected it, got all the
dependencies covered.  Then trying to install it ... that's where
things go seriously sideways.  It's much newer (from Debian 10),
whereas linuxmafia.com is on what looks to be mostly a mix of
Debian 5 and 6.  The format of the .deb is too new.  Tried working
around it, but no real way to do that and have it play nice with the
package management system.

Also tried doing it step-wise on relevant versions - basically run into
same snags.  Try to do very minimal upgrades on Debian to dpkg and other
bits as needed to work around that issue ... no dice.  Far too many
dependencies to do that without breaking tons of other stuff, or majorly
upgrading.

So, I figure ... well, a mostly 5/6 system is hazard in a lot of ways,
and really ought get upgraded or replaced anyway, so ...
now, fresh install, and trying to merge in the tons of customizations
on the host, that would be quite the pain.  So, I figure try the
setp-wise upgrades.  Can upgrade from one Debian major version to next.
So ... from 5/6, to 6, 7, 8, 9, 10.

And, going that route ... first I was trying to preserve all the
customizations ... feasible, but a lot of work/time at each version
upgrade.  So, after a while, I'm like, yeah, let's mostly skip that for
now, just see if we can get there - if that works, then could
reconfigure for the customizations later.  After all, this one's not
production.

And ... that mostly worked ... major pain points along the way, but
mostly worked.  Oh, and also, though Debian still well supports 32-bit
("i386"), the writing is generally on the wall, don't know if/when
Debian will drop that (probably at some point), so, also along the way,
cross-graded from i386 to amd64 (Debian's name for the applicable
64-bit).  One of the biggest pain points, drive storage capacity and
filesystem management.  Both for guido (physical host) and linuxmafia2
(and will be likewise for linuxmafia.com).  Yeah, sure, filesystems
direct on partitions, or partitions+md (mdadm), nice and simple ...
until you run out of room.  And sure, always keep /boot like that and
damn simple - just size it quite adequately but not excessively, and
that space should hold one likely for a decade or more.  But for
everything else, ought have something better.  Personally, I've been
using LVM ... geez, since ... 1995, ... was using LVM on UNIX (same
initial code base) even long before I was using LVM on Linux ... and it
basically just works.  All my years (decades) of dealing with LVM on
UNIX and Linux, only once have I ever hit an actual bug/issue on LVM -
and it was pretty trivial and easy to deal with and work around (and
that bug also got fixed in fairly short order - and I was also doing
something pretty dang atypical at the time to bump into that bug, so I
was more likely to find some bug out there).  So .. back to linuxmafia2
... as I've been bumping into space issues, I've been changing
partitions to use LVM, and dealing with it that way (there were a few
partitions that were unused, so started pressing those into service,
while also trying to be conservative on space usage, notably as guido
doesn't have a whole lot 'o space to spare).  Then we get to guido and
its space.
$ hostname && (cd /sys/block && grep . sd[ac]/size)
guido
sda/size:250069680
sdc/size:250069680
$
Those are its primary drives, essentially everything between the two
done RAID-1 (with md), so total usable space
250069680/2/1024/1024
< 120 GiB
That's not a lot to work with.  My very first SSD from 2011 is 150G
(149GiB).  I think guido - hosting itself, and all VMs thereupon
(notably linuxmafia.com, and also linuxmafia2 and anything else),
< 120 GiB (and that's before we get to filesystem overhead and such)
really isn't a whole lot to work with.
Of course if it was set up with LVM, would be easier to better utilize
the space ... but ... no LVM or the like, just partitions and md.
So, e.g., things are getting pretty tight on /var (where most of the VMs
storage is) on guido.  There's some space on other filesystems, but
since no LVM, and just partitions and md and growing is really hard,
they were sized to likely be adequate for quite a while ... which also
means, that in the meantime, anything over provisioned (thus far) there
can't really be put to use elsewhere.  Whereas with e.g. LVM, provision
without as much space, leave the excess spare ... need a bit more, just
grow LV and filesystem using a bit from the spare unused space.

There are also alternatives, e.g. ZFS - it's very solid, but the
position/relationship (mostly on account of licensing) is bit more
"interesting", and it doesn't have as long a track record on Linux - but
mostly looks quite good (I've certainly also used it quite a bit on
Linux).  But ZFS is a very different animal - so quite the learning
curve.  There's also btrfs, which is quite popular and gaining in use
... but it's also the relatively new kid on the block, so I wouldn't be
inclined to jump on it too soon for something as critical as
filesystems.

Anyway, some other pain points, but mostly got linuxmafia2 upgraded to
Debian 10 with mailman 1:2.1.29-1+deb10u5 so that looks like a very
doable path for linuxmafia.com - but would be rather long and tedious,
notably for prod would want to integrate each of the customizations for
all the applicable software with each upgrade - so that makes it kind of
a pain, ... but want to keep production up as much as feasible.  For
proof-of-concept I skipped lots of that - notably didn't properly and
carefully follow all the recommended steps in upgrade procedures, mostly
picked maintainers version of new configuration files with few
exceptions (notably GRUB and watchdog), and went with that.

I'm also inclined to redo the virtual hardware.  Originally it was set
to be as similar as ye olde police siren physical.  But that probably
matters less and less as time progresses.  Oh, also on linuxmafia2 did
upgrade the virtual CPU and RAM before going to 64-bit.  I did manage to
have it crash but I think only twice, under very heavy network+disk I/O.
Seems not to impact guido at all, but something with(in) the VM.  I
think a lot of the less common virtual hardware configurations plus
pushing that real hard on I/O may quite correlate to the crashes ... and
that may be what sometimes gives us our crashes on linuxmafia.com.  A
much more mainstream virtual hardware configuration may cause that issue
to go away ... and that's one of the next things I'm intending to test
in linuxmafia2.

There are some other minor pain points but most or all of them have to
do with some older software and their configurations.  Thus far in
upgrading I've generally not gotten rid of older obsolete software that
it didn't require be gone (e.g. if it didn't conflict with the newer, it
generally still remains).  A couple such packages and their
configurations have become annoying, and if they can be gotten rid of,
that would clear their annoyances ... and one other package I just need
to get the configuration right (or may need to just get relevant helper
utility(/ies) installed).

Anyway, at least Debian 10 would be a supported platform ... though it's
LTS support only runs through 2024-06-30.  And Debian >=11 doesn't
support mailman 2.x, so longer term, other solutions will be needed
(mailman 3.x or some other list software).  BALUG.org is in the same
boat in that regard - it's still running mailman 2.x on Debian 10.
So ... will need to find solution(s) in the coming month(s) for that
too.

Oh, and LVM would make snapshotting data much more feasible.  At present
that's mostly not very available (e.g. break the RAID-1 to go non-RAID
to get a snapshot, grab that data, then remake the RAID-1 mirror - far
from ideal way to get a snapshot).  ZFS has some bloody amazing snapshot
capabilities ... but like I say, ZFS, whole different animal.

On Sat, Mar 2, 2024 at 1:26 PM Rick Moen <rick at linuxmafia.com> wrote:
>
> Quoting Michael Paoli (michael.paoli at berkeley.edu):
>
> > The one key difference of relevance, at least that I can easily spot,
> > between 2.1.29 and 2.1.30 is:
> > there is now a dmarc_moderation_addresses
> > list setting that can be used to apply dmarc_moderation_action to mail
> > From: addresses listed or matching listed regexps. This can be used
> > to modify mail to addresses that don't accept external mail From:
> > themselves.
>
> You know, it looks to me like 2.1.29 is a 95% solution to the problem,
> so I'd be grateful to have the first while pondering whether 2.1.30 is
> feasible -- or justifiable.  The fact that 2.1.29's the last
> Debian-packaged version (in all likelihood) is a powerful argument
> against venturing past that without a lot of careful thought.
>
> I don't know of specific disadvantages to 2.1.30; I do know that it's in
> production on major sites like listman.redhat.com.  (Naturally,
> using RHAT packaging.)
>
> Years ago, I looked into the change history of when Mailman 2.1.x
> gradually improved the DMARC mitigation over several releases, but
> don't have that work in front of me.  I'll try to find the time to
> reacquire that knowledge.
>
>
> _______________________________________________
> conspire mailing list
> conspire at linuxmafia.com
> http://linuxmafia.com/mailman/listinfo/conspire