[conspire] Slice of life

Thu Sep 17 19:23:14 PDT 2009

At $FIRM, nothing may be changed in the production environment without
approval of a formal change control proposal.  Part of what I do
involves managing DNS and domain registration for the firm's (at last
count) roughly 1840 Internet domains, not counting reverse DNS.

Here's a typical change control request for DNS that I banged out a few
minutes ago.  (Yes, still using cvs in 2009.  Sad, I know.)
"example.com" gets cited, here, in place of the real domain name.

The "named-conffile" test used is something I invented to compensate for
BIND9's notorious tendency to die with no usable diagnostic information
whenever there is _any_ syntax problem in its conffiles or any DNS zonefile.
It's now considered mandatory, here, for all DNS changes, and locally termed
"the Rick test".  (Having to figure out why BIND is dying on startup on
the master nameserver, with no clues, under time pressure, is no fun
whatsoever.)

Point is:  Any operation that's serious about quality and process
control does this, i.e., everything's in a VCS, every change has a
change window and pre-scripted backout procedure, and the live
public environment is never "hotfixed", but rather receives planned
change pushes from the development environment, accompanied by testing
them.  If anything goes wrong, either I or anyone else can run the
backout procedure and immediately revert changes.

This is also yet another reason why shell scripting, sed, awk, find,
grep, xargs, perl, etc. persist over crappy GUI toolsets:  They permit
relatively writing in advance of precise, tested change routines with
matching backout procedures.  I can thus edit 600 domains at once, as
easily and precisely as I can two.

Notice that the backout procedure checks the previous zonefile revisions
out of cvs and then bumps their serial numbers _twice_ each.  Anyone
who's managed DNS is nodding, right now, because the iron rule of
zonefile S/Ns is that they must always go only upwards.

 Subject: Change Control Request: Sunfire and Netapp DNS
 Date: Thu, 17 Sep 2009 18:34:38 -0700
 From: Rick Moen 
 To: [operations mailing list], [colleague]

*** Needs Assessment -

[Colleague] recently deployed a Sun Microsystems SunFire X4540 in [data
centre] as a network storage device with hostname ki5-18.example.com =
IP 10.112.2.165.  He's reported some problems, however, because that
IP's reverse DNS (PTR entry) doesn't match, resolving to hostname
ki33-29.example.com, instead.

Also, [colleague] needs new DNS entries for netapp121.example.com
= IP 10.22.0.121 and netapp122.example.com = IP 10.22.0.122.

Reference:  RT-133005

*** New Features -

Correct one rDNS entry, add two new forward entries.

*** Assumptions -

None.

*** Risks -

Low.  This is a known procedure.

*** Process/Procedure -

#In my sandbox:
cd ~/cvs/site/confs/named/master

Add to zonefile e/example.com-cage, in the netapp section:
; Added for RT-133005 - rmoen 20090917
netapp121       IN  A   10.22.0.121
netapp122       IN  A   10.22.0.122

sed -i 's/ki33-29/ki5-18/' reverse/2.112.10.in-addr.arpa

#Update S/N:
/site/bin/ops/serial.sh e/example.com-cage
/site/bin/ops/serial.sh reverse/2.112.10.in-addr.arpa

# Double-check changes:
cvs diff e/example.com-cage
cvs diff reverse/2.112.10.in-addr.arpa

cvs ci -m "Add 2 example.com entries, fix ki5-18 rDNS - RT-133005" \
e/example.com-cage reverse/2.112.10.in-addr.arpa
merge_patcher \
e/example.com-cage reverse/2.112.10.in-addr.arpa
sudo r2qa -u -q devel,staging,20090901 \
e/example.com-cage reverse/2.112.10.in-addr.arpa
sudo p2c --cluster=admin \
e/example.com-cage reverse/2.112.10.in-addr.arpa

# Reload zones:
rndc reload

# Check /var/log/messages for errors.

*** Change Window -

When convenient.

*** Test Plan -

dig netapp121.example.com. @localhost +short
dig netapp122.example.com. @localhost +short
dig -t ptr 165.2.112.10.in-addr.arpa. @localhost +short
# Should return, respectively, 10.22.0.121, 10.22.0.122, and
# ki5-18.example.com.

*** Back out Procedure -

cd ~/cvs/site/confs/named/master/

cvs up -r 1.341 -p e/example.com-cage > e/example.com-cage
cvs up -r 1.3 -p reverse/2.112.10.in-addr.arpa >  reverse/2.112.10.in-addr.arpa
#Update S/Ns:
/site/bin/ops/serial.sh e/example.com-cage
/site/bin/ops/serial.sh e/example.com-cage
/site/bin/ops/serial.sh reverse/2.112.10.in-addr.arpa
/site/bin/ops/serial.sh reverse/2.112.10.in-addr.arpa
# Double-check changes:
cvs diff e/example.com-cage
cvs diff reverse/2.112.10.in-addr.arpa
cvs ci -m "Reverting RT-133005 changes" \
e/example.com-cage reverse/2.112.10.in-addr.arpa
merge_patcher \
e/example.com-cage reverse/2.112.10.in-addr.arpa
sudo r2qa -u -q devel,staging,20090901 \
e/example.com-cage reverse/2.112.10.in-addr.arpa
sudo p2c --cluster=admin \
e/example.com-cage reverse/2.112.10.in-addr.arpa

# Reload zone:
rndc reload

# Check /var/log/messages for errors.
dig netapp121.example.com. @localhost +short
dig netapp122.example.com. @localhost +short
dig -t ptr 165.2.112.10.in-addr.arpa. @localhost +short
# Should return, respectively, null, null, and
# ki33-29.example.com.

*** Approval -

Pending:  Another Sr. SA, [Manager].