[conspire] Sat, 1/10 Installfest/RSVP

[Rick Moen]

Quoting Nick Moffitt (nick at zork.net):

> Note that I never said that I specifically *need* an ABI.  I first said
> that it wasn't an unreasonable desire, and then that I find it useful
> for ensuring that multiple groups of people are all on the same page
> with a minimum of communication: "Make it work on Hardy or it doesn't go
> out."  Sometimes what you need is to remove variables in as many places
> as you can.  I don't see this as "foolishness".
 
Making it work on Hardy in the sense that it be built by typing
"fakeroot dpkg-buildpackage" (or whatever variant on that you use
locally) should also suffice to make it successfully build, install, and
run in any other binary environment.  I mean, that's part of what we
have self-hosting distributions and dependency-tracking _for_.

> [...] A change in the command-line options to a program can make an
> important shell script start spewing garbage and spawning an unholy
> army of the dead.  This sort of thing will bite anyway, but better
> that it happen during a scheduled upgrade than randomly on a weekly
> walk down dist-upgrade lane.

Yes, yes.  A carelessly built upgrade or a poorly documented fundamental
change can bite you.  I didn't figure out for quite a while that
Deirdre's PHP4 code for BALE had broken because MySQL suddenly ceased
working without explicit COMMIT statements between one release and
another, and I somehow missed any warnings about that, that might have
been present.

So, once in a very long time, you get unexpected breakage following an
upgrade, and have to chase it down.  I just haven't yet seen this be a
big deal.

> I think that this an important difference here.  A few dozen boxes are
> about two or three racks' worth, I suppose.  One person could reasonably
> keep track of that without even resorting to written records for very
> much.

If I had hundreds of machines and an obligation to extremely high levels
of availability, I'd probably use something like the golden master
system to checkpoint changes -- or at least a private flow-through
repository on which I did some local QA, filtering upstream.



> Even on my private mailman installation (which hasn't been even
> moderately busy since the free-sklyarov list shut down) the amount of
> spam that accumulates in the qfiles, blocking my upgrades, is
> depressing:
> 
> [nick at frotz(/var/lib/mailman/qfiles)] for i in *; do echo -en "$i:\t"; sudo ls -1 $i/ | wc -l; done
> archive:        0
> bounces:        36
> commands:       0
> in:     8
> news:   0
> out:    0
> retry:  286
> shunt:  3
> virgin: 0
> 
> The retry queue is the real killer, usually (especially after an
> "unshunt").  I've taken to just covering my eyes, saying "It's all just
> spam, right?  Right?" and blasting it before resuming the upgrade.  I
> feel reasonably comfortable doing that on my private server.  Production
> mailing lists for an important service?  Not quite so much.  

This is what I routinely see:

linuxmafia:~# cd /var/lib/mailman/qfiles/
linuxmafia:/var/lib/mailman/qfiles#  for i in *; do echo -en "$i:\t"; ls
-1 $i/ | wc -l; done
archive:        0
bad:    0
bounces:        0
commands:       0
in:     0
news:   0
out:    0
retry:  0
shunt:  0
virgin: 0

> If you have some magic anti-backscatter system that keeps
> conspire-bounces from hanging on to hundreds of undeliverable "Your
> mail: V1 at kruh n0w! is being held for moderation" messages and
> "Unrecognized command: satisfy ur womn" output, I'd love to hear it!

Nothing all _that_ special.  I make admin queues time out pretty quickly
on the Mailman end.  On the MTA end, Exim4 checks SPF RRs, does callouts
to ensure RFC-compliance of various sorts, runs SA during the incoming
SMTP session to measure spamicity before accepting the mail, and so on.

There is some backscatter, to be sure, but yes, a whole lot less than on
your system.