[conspire] DNS & nameservers ...

[Rick Moen]

Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):

> Well ... sort of kind of almost.  ;-)  TTL gives the *maximum* time (in
> seconds) the RR will be cached (or actually, even more specifically,
> is to be considered valid).  Nameservers *may* discard the data
> sooner ... and many may do so, e.g. due to memory constraints/pressure,
> or some other configuration bits - such as capping caching to some
> particular max TTL time - or may use or combine such with
> Least Recently Used (LRU) algorithms - favoring retention longer (up
> to TTL) for things that are getting frequently or semi-frequently
> referred to, and likewise disfavoring things that just aren't getting
> repeated queries on any frequent basis.  Of course, too, where memory
> is plentiful, there's often much less (to about zero) incentive to drop
> things early/earlier from cache.

Yes yes yes yes yes.  But there you go awain, gnawing away at the fine
points and edge-cases while obscuring the larger and more-significant
point I was trying to make -- which was to explain for the benefit of
newcomers why short TTLs are sometimes helpful in some cases and long
TTLs in others.

> Alas, I haven't (yet) dropped in such automated monitoring 8-O ... but I'm
> typically mucking about in DNS semi-regularly enough I typically end up
> checking it about a few times per month or so anyway, and certainly at
> bare minimum multiple times per quarter ... so that's at least partial
> coverage (yes, do still have on my long todo list to add some better and
> more complete coverage on monitoring, and other automation.

The perfect is the enemy of the good-enough, so I'm entirely happy with
my slapdash but adequate weekly cronjobs to check up on various domains
of interest to me and their DNS quality.  Example:

:r /etc/cron.weekly/mydomains

#!/bin/sh

# mydomains     Cron script to sanity-check my domains' SOA records at
#               all of their authoritative nameservers, as a quick and 
#               dirty way of making sure (1) they're all online and
#               (2) they're all serving up the same data (or at least
#               data with the same zonefile serial number).
#  
#               The script queries all nameservers for their current
#               SOA value, and then uses awk to parse out of that 
#               verbose record just the S/N field, which is field #3.  
#               The point is that you can visually spot offline or 
#               aberrant nameservers by their S/Ns being (respectively) 
#               missing or an out-of-step value.
#
#		Written by Rick Moen (rick at linuxmafia.com)
#		$Id: cron.weekly,v 1.03 2011/05/21 00:35:00 rick
# Copyright (C) Rick Moen, 2011.  Do anything you want with this work.

set -o errexit  #aka "set -e": exit if any line returns non-true value
set -o nounset  #aka "set -u": exit upon finding an uninitialised variable

test -x /usr/bin/mail || exit 0
test -x /usr/bin/whois || exit 0
test -x /usr/bin/awk || exit 0
test -x /bin/grep || exit 0
test -x /usr/bin/dig || exit 0


{
echo "As of 2011-05-21, linuxmafia.com should show five authoritative nameservers:"
echo ""
echo "ns.primate.net. 198.144.194.12, (Aaron T. Porter)"
echo "ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)"
echo "ns3.linuxmafia.com. 63.193.123.122, aka ns.catwhisker.org (David Wolfskill)"
echo "ns1.thecoop.net. 66.220.20.163, (Drew Bertola)"
echo "ns1.linuxmafia.com. 198.144.195.186 (Rick Moen)"
echo ""
echo "As of 2011-05-21, unixmercenary.net should show five authoritative nameservers:"
echo ""
echo "ns.primate.net. 198.144.194.12, (Aaron T. Porter)"
echo "ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)"
echo "ns3.linuxmafia.com. 63.193.123.122, aka ns.catwhisker.org (David Wolfskill)"
echo "ns1.thecoop.net. 66.220.20.163, (Drew Bertola)"
echo "ns1.linuxmafia.com. 198.144.195.186 (Rick Moen)"
echo ""
echo "If any is missing from reports below, or produces odd data, something is wrong."
echo ""
echo "Zonefile S/Ns, linuxmafia.com:"
echo ""
dig -t soa linuxmafia.com. @ns.primate.net. +short | awk '{ print $3 " on ns.primate.net." }'
dig -t soa linuxmafia.com. @ns.tx.primate.net. +short | awk '{ print $3 " on ns.tx.primate.net." }'
dig -t soa linuxmafia.com. @ns3.linuxmafia.com. +short | awk '{ print $3 " on ns3.linuxmafia.com." }'
dig -t soa linuxmafia.com. @ns1.thecoop.net. +short | awk '{ print $3 " on ns1.thecoop.net."}'
dig -t soa linuxmafia.com. @ns1.linuxmafia.com. +short | awk '{ print $3 " on ns1.linuxmafia.com."}'
echo ""
echo "Zonefile S/Ns, unixmercenary.net:"
echo ""
dig -t soa unixmercenary.net. @ns.primate.net. +short | awk '{ print $3 " on ns.primate.net." }'
dig -t soa unixmercenary.net. @ns.tx.primate.net. +short | awk '{ print $3 " on ns.tx.primate.net." }'  
dig -t soa unixmercenary.net. @ns3.linuxmafia.com. +short | awk '{ print $3 " on ns3.linuxmafia.com." }'
dig -t soa unixmercenary.net. @ns1.thecoop.net. +short | awk '{ print $3 " on ns1.thecoop.net."}'
dig -t soa unixmercenary.net. @ns1.linuxmafia.com. +short | awk '{ print $3 " on ns1.linuxmafia.com."}' 
echo ""
echo "Authoritative nameservers from whois, linuxmafia.com:"
echo ""
whois linuxmafia.com | grep 'Name Server' | awk -F: '{ print $2 }' | head -n 7
echo ""
echo "Authoritative nameservers from whois, unixmercenary.net:"
echo ""
whois unixmercenary.net | grep 'Name Server' | awk -F: '{ print $2 }' | head -n 7
echo ""
echo "Parent-zone NS records and matching A records (glue), linuxmafia.com:"
echo ""
dig -t ns linuxmafia.com. @$(dig -t ns com. +short | head -n 1) +nocmd +noquestion +nostats +nocomments
echo ""
echo "Parent-zone NS records and matching A records (glue), unixmercenary.net:"
echo ""
dig -t ns unixmercenary.net. @$(dig -t ns net. +short | head -n 1) +nocmd +noquestion +nostats +nocomments
echo ""
echo "In-domain NS records and matching A records, linuxmafia.com:"
echo ""
dig -t ns linuxmafia.com. @$(dig -t ns linuxmafia.com. +short | head -n 1) +nocmd +noquestion +nostats +nocomments
echo ""
echo "In-domain NS records and matching A records, unixmercenary.net:"
echo ""
dig -t ns unixmercenary.net. @$(dig -t ns unixmercenary.net. +short | head -n 1) +nocmd +noquestion +nostats +nocomments

} |
mail -s "Domains linuxmafia.com and unixmercenary.net SOA check" rick at linuxmafia.com




> Also of note, many software monitoring packages have excellent
> capabilities to monitor DNS ... but that may also be overkill,
> depending upon one's objectives).

> In this particular case I was updating zones
> (for letsencrypt.org validation by DNS(+DNSSEC)), and wanted to ensure that
> all the authoritative nameservers had picked up the new data ... SOA check
> alone didn't *fully* test that, but was a "good enough" test

More than good enough.  For a domain's auth nameservers to have the same
SOA S/N subfield value but nonetheless different zone contents is such a
farfetched failure scenario as to be not worth worrying about.

Remember:  The perfect is the enemy of the good enough.  Never get
caught up in being clever and take your eye off the ball.

> 
> Oh, ... also good to note and save contact information for DNS nameservers,
> preferably including multiple means, e.g. email, phone, alternate contact
> means/numbers/addresses, etc. - and store them somewhere they can be found
> years later ... like as comments in zone master files (or for slaves, as
> comments in the relevant configuration files).

I do it in the latter.

Examples of that, redacted to remove telephone numbers and other data
deemed too private, is in my example BIND9 archive, 
http://linuxmafia.com/pub/linux/network/bind9-examples-linuxmafia.tar.gz

> Some of us remember drives, and terminating resister packs.  Sometimes
> you needed 'em on (end of chain), sometimes not (not end of chain)....

Oh, that one is simple.  If you were not using a resistor pack, (as you
say) you fastened it using strong tape and/or a small ziplock bag to the
device it pertains to.  You _never_ let it get separated and trust to
your ability to re-find the correct pack when it's needed in the future.
That would qualify as a dumb planning failure.


> Well, nowadays, to *some* extent, ... that kind'a depends.

No, it really doesn't depend,

Five reliable, network-dispersed authoritative nameserver domains is
better than four, which is better than three.

Seriously. Michael, why are you always wanting to play Edge-Case Theatre?

> There are various ways to do, e.g. high availability with network
> stuff and routing even with a single IP. 

Reckless, because SPoF, even if you try to clever-away your basic error
using HA, etc.

But again, my God, Michael, I gave a simple solution to increase
reliability for any domain owner contemplating authoritative DNS, one
that works and is cheap, effective, and uncomplicated, and your response
is to say 'Well, you could in _theory_ get all of those advantages using
only a single IP address if you use insanely complicated HA methods.'

Really, Michael?  You seriously think that is a rational comment in
context?


> But yeah, DNS, ... don't do *just* a single IP for authoritative
> nameserver, ... because, well, Mr. Murphy and his laws are alive and
> well and often quite practiced. 

Nor two, either.

RFC recommendation is minimum three, maximum seven.  This is hard-won
wisdo from experts, and only an idiot (IMO) decides 'I don't care; 
I think two is enough.'


> So, a highly redundant single IP won't do diddly for ya' when that one
> IP still fails anyway.  So yes, minimum two. 

No, goddamn it.  I already explained why not.

Do you think you know about DNS reliability better than the authors of
the relevant RFCs on the subject?  No?  Then, you shouldn't blithely 
decide to ignore their recommendations.

And I would really appreciate it if you stopped giving people bad
advice.  Seriously.

> Beyond that ... really how important and what's your risk model ...
> usually three (or more is good).

Not 'usually'.


> "Too many" can also be problematic.  And three (to five or a bit more)
> that are rock solid (or nearly so), is often significantly better than
> five or six or seven or more with several of 'em being sort'a kind'a
> mostly available except when too frequently they're not for some fair
> bit.

Move the goalposts much?

Six reliable auth nameservers are better than five reliable auth
nameservers.  That is what I was saying.  Your moving the goalposts and
wanting to compare three-to-five reliable nameservers with six or seven
that includes unreliable ones is tiresome gamesmanship, and wastes your
time and mine.

How do you determine whether your auth nameservers are reliable?
Monitoring, obviously, which is the correct answer irrespective of how
many.  Above, I inlined a perfunctory weekly cronjob that is my
current good-enough solution for that.  Much better can be achieved with
some work.  (Feel welcome to do it.)