[sf-lug] Problems of being a backup MX (was: Warning: message 1O1ezP-0004Uk-FA delayed 48 hours)

[Rick Moen]

Lx Rudis asked me an offlist question this morning.  I gave him a quick
if sketchy answer, while promising to write a proper, full answer later.
It took a while to write, so I thought I'd post it here for collective
knowledge.

Quoting Lx Rudis (lx_rudis at sbcglobal.net):

> hey rick, i figure you're seeing the attached email also but i thought i should ask you directly about it.
> has sflug been compromised by a spammer?  
> is there anything _i_ am remiss in doing about this?
> 
> 
> lx
> 
> 
> 
> 
> ----- Forwarded Message ----
> From: Mail Delivery System <Mailer-Daemon at linuxmafia.com>
> To: sf-lug-owner at linuxmafia.com
> Sent: Thu, April 15, 2010 5:20:50 AM
> Subject: Warning: message 1O1ezP-0004Uk-FA delayed 48 hours
> 
> This message was created automatically by mail delivery software.
> A message that you sent has not yet been delivered to one or more of its
> recipients after more than 48 hours on the queue on linuxmafia.com.
> 
> The message identifier is:     1O1ezP-0004Uk-FA
> The subject of the message is:  Discount  54%
> The subject of the message is: Discount  54%
> 
> The address to which the message has not yet been delivered is:
> 
>   duncan at randometry.org
>     (generated from installers at linuxmafia.com)
>     Delay reason: SMTP error from remote mail server after RCPT TO:<duncan at randometry.org>:
>     host zork.net [70.85.129.199]: 450 4.1.1 <duncan at randometry.org>:
>     Recipient address rejected: unverified address:
>     connect to spin.randometry.COM[66.93.140.140]:25:
>     Connection timed out
> 
> No action is required on your part. Delivery attempts will continue for
> some time, and this warning may be repeated at intervals if the message
> remains undelivered. Eventually the mail delivery software will give up,
> and when that happens, the message will be returned to you.



OK, I finally have time to unpack this stuff for you.  In the Big
Inning, as the old baseball joke goes (i.e., 'in the beginning')
my server's MTA address, Mailer-Daemon at linuxmafia.com, for some reason
(1) was trying to send mail to "installers at linuxmafia.com", but also 
(2) decided to inform "sf-lug-owner at linuxmafia.com" about the results
of that effort:

> The address to which the message has not yet been delivered is:
>
>   duncan at randometry.org
>     (generated from installers at linuxmafia.com)
[...]
> From: Mail Delivery System <Mailer-Daemon at linuxmafia.com>
> To: sf-lug-owner at linuxmafia.com


It's perfectly obvious why Mailer-Daemon at linuxmafia.com was attempting
to get mail to installers at linuxmafia.com:  "installers" is a heavily
advertised e-mail address, that is shown on CABAL's Web pages and
announcements as a point of contact to, e.g., let us know that you're 
coming to CABAL events and need special help with a problem-child
computer, or that you might want us to have on hand a relatively rare
Linux or BSD distribution, or something like that.

Being heavily advertised, the "installers at linuxmafia.com" address is
continually subjected to spam attempts from all over the world.  That is
not per-se a problem but rather an unavoidable consequence of actually
bothering to be reachable on the Internet.

Before anyone asks, no, I absolutely do not favour hiding addresses from
spammers.  I am not going to hide from them.  They might at some point
need to hide from me, but it's my Internet, I live here, and I do not
hide from petty criminals.

I have no idea offhand why Mailer-Daemon at linuxmafia.com sought to brief
sf-lug-owner at linuxmafia.com about results of attempting to deliver mail
to installers at linuxmafia.com.  There would seem no connection between
the sf-lug at linuxmafia.com mailing list and "installers".  If I have time
and interest, I might investigate, but definitely not today.

sf-lug-owner at linuxmafia.com is an internal Mailman mail alias that
resolves to both jim at well.com and lx_rudis at sbcglobal.net.  I.e., it is a
public address that reaches you both in your organisational capacity as
listadmins of the sf-lug at linuxmafia.com mailing list.

"installers at linuxmafia.com" is a regular, non-Mailman e-mail alias,
defined in my system's /etc/aliases file.  It likewise resolves to a
pair of addresses:  rick at linuxmafia.com and duncan at randometry.org .

The second of those addresses is that of Duncan MacKinnon of San
Francisco, co-head with me of CABAL.

These past few years, Duncan has had a problem:  He's been so incredibly
busy with the rebuild of his house in Sunset Heights that his server has
frequently been offline for long periods at a time.  In fact, last I
heard, he really would appreciate my just building a new server for him,
which I've been meaning to do, except that things come up and sap my
attention (surgery, and like that).  I still want to build him that
server.  It's on my TO DO list.  It'd get done sooner if I didn't come
home and think:  I don't want to see any more computers today.  Maybe
some gardening.  Maybe a hike or bicycle ride, pet the cats, cook a nice
meal.  Also, I have kind of a backlog of projects, anyway.

So, what happens when your mail server is down?  Let's say you have a
machine that is declared in your domain's DNS as an MX-type (Mail
eXchanger) entry.  It goes offline.  Let's say Duncan's main server for
randometry.org does that.

Other machines trying to send it mail encounter failure when trying to
open the socket to port 25.  What do they do next?  They go back and
look through te destination domain's DNS for _other_ "MX" entities -- 
ones with higher-value and thus lower-priority metrics that are supposed
to show which to consider primary and which fallback.  Let's see what
we're talking about, specifically for the randometry.org domain:

:r! dig -t mx randometry.org +short

10 spin.randometry.COM.
20 zork.net.

So:  There are two MX entries in that domain's (randometry.org's) DNS.
The first one, priority 10, is spin.randometry.COM.  That's Duncan's
personal server, the one that is often offline.  The fact that the "10"
value is the lower of the two that exist means this MX is _primary_,
i.e., that delivering MTAs are asked to please try spin.randometry.COM
_first_, and zork.net only if spin.randometry.COM is unreachable.

In the situation under discussion, spin.randometry.COM _is_ in fact
unreachable, so the delivering MTA (linuxmafia.com, in this discussion)
attempts to drop off that mail at zork.net, which is a server run by my
friend Nick Moffitt.

What's going on there?  Years ago, Duncan asked Nick, 'Nick, any chance
you would be willing to have zork.net serve as a my fallback MX?'  Nick
said 'sure', and all was well for some years.  But eventually, Nick
called or wrote to Duncan, saying 'I'm re-doing my mail system, and
disabling its willingness to accept mail addressed to randometry.org.
You'll want to update your DNS accordingly.  Unfortunately, Duncan's
been so busy that he's not removed the "20" MX from his DNS, so delivery
attempts to zork.net persist every time Duncan's own mail server is
offline.


And zork.net says:

    host zork.net [70.85.129.199]: 450 4.1.1 <duncan at randometry.org>:
    Recipient address rejected: unverified address

The "450" is an SMTP temporary failure code.  Permanent failures are
55x.  Tempfails are often implemented by sysadmins as a form of
'teergrubing' (tarpitting) to screw up attempts to deliver spam.  The
idea is that you're punishing the remote MTA attempting to drop off spam
at your MTA by not refusing it outright (550 or 552), but rather saying
the SMTP equivalent of 'Not now, but you're perfectly welcome to try
again later.'  The idea is to waste the spamhaus's time.  (There are
disadvantages -- beyond the scope of this mail.)

Past doubt, this _is_ spam.  You can tell by the Subject headers.
(Apparently, there are two mails in queue.)

> This message was created automatically by mail delivery software.
> A message that you sent has not yet been delivered to one or more of its
> recipients after more than 48 hours on the queue on linuxmafia.com.
>
> The message identifier is:     1O1ezP-0004Uk-FA
> The subject of the message is:  Discount  54%
> The subject of the message is: Discount  54%


Nick is presumably a bit pissed off that Duncan _still_, after some
years, hasn't bothered to update his DNS to stop trying to drop off
randometry.org's mail at zork.net as a relay -- years after Nick said
"Duncan, I'm ceasing to be your relay."


This whole syndrome is one of the reasons why it's really pretty rare
for sysadmins to do the courtesy of being backup MXes for each other,
any more.  Over years, secondaries forget and shut off the relaying
service without quite intending to (and without notice), or the primary
leaves a machine offline for a long time, such that secondaries start
building up huge mail spools that will later need redelivery to the
primary.  Also, the spammers long ago figured out that SMTP
lower-priority MXes often have _much_ more lax antispam defences than do
the primaries, so they _deliberately_ violate the RFCs by preferentially
ignoring the primaries and dropping mail at the high-numbered
(low-priority) mail hosts -- which then work frantically to redeliver to
the primary.  At best, you end up with the secondary triggering the
primary's antispam defences during attempts to redeliver the spool, to
the extent that the spooled mail includes detectable spam, which will
invariably be the case.  That's much like what Nick's zork.net is in
fact attempting on my machine during the latter's delivery attempt
(teergrubing).

Because of those and other problems, domains doing backup MX for each
other has been tending to go the way of the dodo, except in corner cases
such as where a single admin controls both domains and can ensure that
all machines implement the exact same antispam policies/measures.

So, for example, my own domain has _no_ backup MX:

:r! dig -t mx linuxmafia.com +short

10 linuxmafia.com.

That means that, if my mail server goes down, I have a couple of days to
deploy a replacement mail system somewhere on the Internet, and
re-point my DNS to it, or mail will start bouncing.  I figure I can do
that -- and have:  Last April, we had a big thunderstorm that fried
almost all of the then-existing linuxmafia.com hardware, which was a
PIII/500 with 256 MB RAM and a pair of 9 GB SCSI hard drives.  Within
about two hours, I had a basic setup of Debian installed and accepting
mail on a replacement box, which is the PIII/800 with 1.5 GB RAM and 73
GB boot SCSI drive plus 18GBx2 mirrored pair SCSI drives that is still
running the thing today.

Why do I keep duncan at randometry.org in the "installers" alias?    Well ,
because Duncan doesn't have his machine offline _all_ the time, and,
besides, it reminds me that I still want to build him a server.

Hope that helps.