[sf-lug] SF-LUG DNS
Michael Paoli
Michael.Paoli at cal.berkeley.edu
Tue Nov 17 00:09:38 PST 2009
My comments, below, in-line
references/excerpts:
> Date: Mon, 16 Nov 2009 08:55:16 -0800
> From: Rick Moen <rick at linuxmafia.com>
> Subject: Re: [sf-lug] SF-LUG DNS
> To: sf-lug at linuxmafia.com
>
> Quoting jim (jim at well.com):
>
>> we replaced the old machine with a new one. the
>> old machine was pretty much a classic unix box
>> (CentOS 4 or 5). the new machine uses debian XEN
>> to implement two virtual machines, one for balug,
>> the other for sf-lug (which is now using the
>> correct ip address).
>
> Um... I hope your mailing list posting (quoted above) wasn't intended to
> close out the list of what I said you need to do. You still need to do
> those things. Please consult my rundown of that matter, which I'll
> re-quote (below) to correct typos.
Yes, definitely still stuff to be done (I keep hoping Jim or someone
else will get SF-LUG.COM. DNS squared away before the secondary expires
the zone ... but if the timing gets too close on that, I plan to correct
it - and in the meantime Jim Stockford and/or other SF-LUG.COM.
systems/DNS administrators can contact me if they need assistance or
have questions).
> I'm sorry if I sound like I'm barking orders at you guys, but you
> shouldn't move master nameservice for a domain to a new IP without
> coordinating closely with your secondaries (which, again, should number
> at least two, not just one). Omitting that coordination is pretty much
> guaranteed to break your DNS, which I'm assuming is deemed A Very Bad
> Thing. sf-lug.com's DNS is currently damaged and dangerously thin. If
> not corrected, is going to fail completely, when the secondary's
> copy of the zonefile hits its expire limit.
Actually, by coincidence, turns out the "new" (substituted) master DNS
server is ... well, will be anyway, on the same IP (host is there, but
last I checked it's not yet serving up DNS nor particularly being DNS
for SF-LUG.COM.).
> In that area, it's kind of a good thing that you're not relying on the
> sf-lug.COM domain very much, compared to sf-lug.ORG, but IMO you should
> use this as a learning experience to help with handling of your
> more-vital domain in the future.
>
>> i'll have to bring up the old machine and find
>> the dns files then copy them to the new machine.
And why would one need to do that? As mentioned off-list (in email sent
Date: Wed, 11 Nov 2009 01:57:47 -0800) and also in:
http://linuxmafia.com/pipermail/sf-lug/2009q4/007146.html
we have:
< Date: Wed, 11 Nov 2009 01:57:47 -0800
< From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
< Subject: FYI, fresh full backup of sf-lug.com. "tower" at ...
< To: <redacted alias>@balug.org
<
< FYI, I've also placed a nice fresh backup of almost* everything*
< from the sf-lug.com "tower" box, at least presently (mounted
< ro,nosuid,nodev) under
< /var/local/balug/backup/0/tower
< on the Silicon Mechanics "vicki" dom0 host.
<
< Also, since UIDs/GIDs between the two systems generally don't match,
< the location is rather protected:
< # ls -ldba /var/local/balug/backup/0
< drwx------ 3 root root 72 2009-11-10 22:57 /var/local/balug/backup/0
< #
<
< *I ommitted stuff under:
< /tmp
< /tmpb
< /bootb
< and also skipped virtual and "empty" (or empty except for lost+found)
< filesystems; I also didn't back up any metadata (e.g. disk partitioning
< information, etc.)
So, ... with a little poking around in there, one can find all the
earlier master configuration bits.
I'll post (separately), in a bit, some information on earlier 2009-02
DNS problem ... fair bit of overlap, so that might be
interesting/informative for folks to look/glance over - especially if
they didn't see it earlier. Glancing that over that makes locating the
relevant file bits even easier, ... but short of that, not too hard to
find them:
//I show my added comments on line(s) starting with //
# ifconfig | fgrep inet\ addr; hostname --fqdn; who am I; id
inet addr:208.96.15.250 Bcast:208.96.15.255 Mask:255.255.255.248
inet addr:127.0.0.1 Mask:255.0.0.0
vicki.sf-lug.com
mpaoli pts/0 2009-11-12 05:14 (198.144.194.236)
uid=0(root) gid=0(root) groups=0(root)
//So, that's the host referenced where I mentioned the quite full backup
//was placed, and we see I'm logged in as mpaoli and running as root (e.g.
//used sudo to become root).
# cd /var/local/balug/backup/0/tower && pwd -P
/var/local/balug/backup/0/tower
//the location of the backup, and checking that we got more-or-less
//where we expected to get to
# mount | fgrep backup
/dev/mapper/vg--balug-backup on /var/local/balug/backup type reiserfs
(ro,nosuid,nodev,noatime)
//just peeking at filesystem and mount options - matches what was
//reported earlier
//so, we look around for some likely init stuff
//here > is PS2, not email quoting
# 2>>/dev/null find etc/*.d -follow \( \
> -name '*named*' -o \
> -name '*bind*' -o \
> -name '*dns*' \) -type f -print |
> fgrep -v -i balug
etc/init.d/named
etc/init.d/ypbind
etc/init.d/winbind
etc/log.d/conf/services/named.conf
etc/log.d/scripts/services/named
etc/logrotate.d/named
etc/rc0.d/K35winbind
etc/rc0.d/K87named
etc/rc0.d/K73ypbind
etc/rc1.d/K35winbind
etc/rc1.d/K87named
etc/rc1.d/K73ypbind
etc/rc2.d/S13named
etc/rc2.d/K35winbind
etc/rc2.d/K73ypbind
etc/rc3.d/S13named
etc/rc3.d/K35winbind
etc/rc3.d/K73ypbind
etc/rc4.d/S13named
etc/rc4.d/K35winbind
etc/rc4.d/K73ypbind
etc/rc5.d/S13named
etc/rc5.d/K35winbind
etc/rc5.d/K73ypbind
etc/rc6.d/K35winbind
etc/rc6.d/K87named
etc/rc6.d/K73ypbind
etc/rc.d/rc0.d/K35winbind
etc/rc.d/rc0.d/K87named
etc/rc.d/rc0.d/K73ypbind
etc/rc.d/rc1.d/K35winbind
etc/rc.d/rc1.d/K87named
etc/rc.d/rc1.d/K73ypbind
etc/rc.d/rc2.d/S13named
etc/rc.d/rc2.d/K35winbind
etc/rc.d/rc2.d/K73ypbind
etc/rc.d/rc3.d/S13named
etc/rc.d/rc3.d/K35winbind
etc/rc.d/rc3.d/K73ypbind
etc/rc.d/rc4.d/S13named
etc/rc.d/rc4.d/K35winbind
etc/rc.d/rc4.d/K73ypbind
etc/rc.d/rc5.d/S13named
etc/rc.d/rc5.d/K35winbind
etc/rc.d/rc5.d/K73ypbind
etc/rc.d/rc6.d/K35winbind
etc/rc.d/rc6.d/K87named
etc/rc.d/rc6.d/K73ypbind
etc/rc.d/init.d/named
etc/rc.d/init.d/ypbind
etc/rc.d/init.d/winbind
//our default run level is?
# grep '^[^#].*default' etc/inittab
id:5:initdefault:
//knowing a bit of ye olde tower box history, we happen to know that it
//should have the proper setup to (re)start the SF-LUG.COM. DNS upon
//reboot, hence we look under ...
# 2>>/dev/null find etc/rc.d/rc5.d -follow \( -name '*named*' -o -name
\> '*bind*' -o -name '*dns*' \) -type f -print | fgrep -v -i balug |
> fgrep -v yp | fgrep -v win
etc/rc.d/rc5.d/S13named
# ls -l etc/rc.d/rc5.d/S13named
lrwxrwxrwx 1 root root 15 2009-11-11 00:31 etc/rc.d/rc5.d/S13named ->
../init.d/named
# ls -l etc/rc.d/init.d/named
-rwxr-xr-x 1 root root 5396 2007-02-07 03:35 etc/rc.d/init.d/named
# fgrep sysconfig etc/rc.d/init.d/named
[ -r /etc/sysconfig/network ] && . /etc/sysconfig/network
[ -r /etc/sysconfig/named ] && . /etc/sysconfig/named
# grep '^[^#]' etc/sysconfig/named
ROOTDIR=/var/named/chroot
//and etc/rc.d/init.d/named gets a bit messy, but in part, we have:
start() {
if [ -n "${ROOTDIR}" -a "x${ROOTDIR}" != "x/" ]; then
OPTIONS="${OPTIONS} -t ${ROOTDIR}"
daemon /usr/sbin/named -u named ${OPTIONS};
//so, looking a bit further ...
# find var/named/chroot -type f -print
var/named/chroot/etc/named.conf.rpmsave
var/named/chroot/etc/rndc.key
var/named/chroot/etc/named.conf
var/named/chroot/etc/localtime
var/named/chroot/var/run/named/named.pid
var/named/chroot/var/named/sf-lug.com
var/named/chroot/var/named/named.zero
var/named/chroot/var/named/named.ip6.local
var/named/chroot/var/named/localdomain.zone
var/named/chroot/var/named/named.broadcast
var/named/chroot/var/named/named.local
var/named/chroot/var/named/slaves/evilmonkeys.com
var/named/chroot/var/named/localhost.zone
var/named/chroot/var/named/.sf-lug.com.swp
var/named/chroot/var/named/sf-lug.com.SAFOLD
var/named/chroot/var/named/sf-lug.com.SAF
var/named/chroot/var/named/named.ca
# fgrep directory var/named/chroot/etc/named.conf
directory "/var/named";
# fgrep -C 4 -i sf-lug.com var/named/chroot/etc/named.conf
masters { 69.17.44.159; };
allow-query { any; };
};
zone "sf-lug.com" IN {
type master;
file "sf-lug.com";
allow-transfer { slaves; };
allow-query { any; };
notify yes;
};
# cat var/named/chroot/var/named/sf-lug.com
$TTL 86400
$ORIGIN sf-lug.COM.
@ IN SOA ns1.sf-lug.com. jim.well.com. (
2007102904 ;Serial
3600 ;refresh period
3600 ;retry period
1209600 ;expire period
10800) ;minimum TTL period
;
IN NS ns1.sf-lug.com.
IN NS ns2.sf-lug.com.
;
IN A 208.96.15.252
IN MX 5 mail.sf-lug.com.
IN TXT "v=spf1 a mx -all"
;
www IN A 208.96.15.252
mail IN A 208.96.15.252
;
ns1 IN A 208.96.15.252
ns2 IN A 198.144.195.186
#
//anyway, that should be sufficient basic data (but not a full how-to)
//on what's needed to get SF-LUG.COM. DNS operational again.
> So, you're saying you retired the old nameserver entirely? And the new
> nameserver is on a different IP and doesn't yet have the old one's DNS
> zonefiles (or presumably its nameserver configuration)?
Approximately, ... totally new (at least to that IP) host, but (mostly
by coincidence) still same IP:
$ (cd ~/.ssh && ssh-add -t 5m id_dsa.sf-lug.com.)
Enter passphrase for id_dsa.sf-lug.com.:
Identity added: id_dsa.sf-lug.com. (id_dsa.sf-lug.com.)
Lifetime set to 300 seconds
$ ssh -ax -l mpaoli 208.96.15.252 'umask 077 && { hostname
> /sbin/ifconfig | fgrep "inet addr"; }'; ssh-add -D
Address 208.96.15.252 maps to 208.96.15.252.servepath.com, but this
does not map back to the address - POSSIBLE BREAKIN ATTEMPT!
sflug
inet addr:208.96.15.252 Bcast:208.96.15.255 Mask:255.255.255.248
inet addr:208.96.15.251 Bcast:208.96.15.255 Mask:255.255.255.248
inet addr:127.0.0.1 Mask:255.0.0.0
All identities removed.
$
After the physical system swap outs, the Xen sflug domU was initially
set up on 208.96.15.251, and another domU on 208.96.15.252 ... but
after a bit, we decided (since we happened to get the same subnet
again) to change that, moving that other domU off of 208.96.15.252, so
sflug could reclaim that 208.96.15.252 IP - thus not needing to change
IP of the SF-LUG.COM. DNS master. I also indicated to Jim Stockford,
that the sflug domU could probably just add the 208.96.15.252, and
continue to hang onto the 208.96.15.251 IP (which it had started with)
- at least until the dust settled - and then 208.96.15.251 could be
relinquished from the sflug domU at a later date (no real rush on that).
Most or all of those particular bits of communication happened off-list.
> Ideally, you would take a zone's master nameservice completely
> down only when you're about to bring up its replacement. You would have
> shortened the TTL values in advance of changeover. You would let your
> secondaries know as soon as feasible about the new IP to point to,
> verify that zone transfers are working, and verify that the NS lines in
> _both_ the served zonefiles and the WHOIS records are updated.
Yes, ... not quite the situation in this case. Even more ideally, the
new master would be up before the old master goes down - slaves could
potentially even be configured with both masters, after the new master
is up, and then old master removed from slaves configuration around the
time of the bringing down of the old master. In any case, not exactly
the scenario we're dealing with here. In our case, same IP for master,
but new host - ideally the new host should have been quite fully ready
to take over as DNS master when it was brought up - obviously that
wasn't (and last I peeked still isn't) the case.
> To be clear: It's OK if master nameservice goes offline. That's what
> secondaries are for. But you want to let your secondaries know what's
> going on, and bring up replacement master nameservice before the expire
> limit.
Well, it *can* be 'okay' ... in this case it's also not so good, because
the master is also an NS for the zone - so that introduces
timeouts/latencies on DNS queries, and reduces redundancy ... but DNS is
designed to withstand that ... and does (at least until slave(s) expire
the zone).
>> probably a poor idea to have a master dns
>> server on the box that the dns points to, yes?
>
> Um, no. Why would that be a problem? I'm sorry, but I don't follow
> your reasoning.
That comment caught my eye earlier, ... perhaps I should have commented
on it or addressed it, but didn't ... anyway, Rick Moen well addressed
that.
> By the way, I find it really, _really_ useful to annotate my nameserver
> configuration file with comments. In particular, I put in whose domain
> it is (full contact data), what host each referenced IP address is, plus
> contact information about the administrators of each nameserver (name,
> e-mail address, and telephone number). That way, you never have the
> "Oh, dear, who are my secondaries and how do I reach them to tell them
> I'm moving the master DNS to a new IP" problem.
Absolutely! Excellent point. I did scrutinize the sf-lug.com. old
master zone file before cat(1)ing it and showing the full output here.
Some of the BALUG.ORG. (at least subdomain) zone files contain bits of
(moderately) sensitive data - that's a reason they're access is rather
well controlled (otherwise I'd have no issues with fully listing their
contents). Those files are well annotated with notes/comments, e.g.
including rather detailed information on contacts for slaves (name,
email, phone, etc.) - at least some of which, I'm presuming not
everyone wants all that information made highly public, thus the
restrictions.
> (1) I cite e-mail address "rick at deirdre.net" (in my wife's domain) to
> avoid having the domain contact be in-band for the domain it concerns.
> That is, it's important that the cited address be reachable even if the
> domain and its underlying hardware and/or network goes offline. So,
> "rick at deirdre.net" is picked because it's handled completely outside the
> domain, machine, and network that serves linuxmafia.com.
Excellent point (I earlier said something quite similar, though we
disagree at least somewhat on some other points).
> (2) I carefully do not cite myself as the sole contact (which would
> create a single point of failure problem), and have someone else as
> Technical Contact.
Yes, good point (I think I also mentioned something like that).
>> i wish i could find a tutorial that succinctly
>> and completely explains the rudiments of setting
>> up dns for and on a linux system: this would
>> summarize the concepts and then specify the files
>> to create and/or modify and finally present
>> several examples to exemplify common use cases.
>
> I published a short article a few years ago in _Linux Gazette_ that
> details how to do every variety of DNS _other_ than master authoritative
> DNS, which is covered only in part. I partially punted on that variety,
> because it would take a long, very intricate article to cover how to do
> it _properly_, i.e., to avoid common mistakes and explain why they are
> mistakes. http://linuxgazette.net/121/moen.html
>
> I hope that is useful, despite my having punted on the details of master
> authoritative DNS.
Reasonably to thoroughly covering properly setting up and operating DNS
masters isn't exactly a short trivial topic area to cover. There
literally are books on the topic. E.g.:
http://oreilly.com/catalog/9780596100575/
Some systems administration books
also have reasonable coverage in chapter(s) on DNS.
> Here's that list of what you should do, to move a domain's master
> namservice to a new IP:
Not quite the scenario here, but still excellent points.
More information about the sf-lug
mailing list