[conspire] Fun with shell

[Rick Moen]

There are a lot of file-handling jobs where "Oh, I'll just do it
manually in vi" seems like a fine idea for a small number of files --
but then the larger file quantity n becomes, the worse an idea it seems.


Let's say you had... oh...

[rmoen at ii92-40 (23:00:00) ~/cvs/site/confs/named/master]$ wc -l filelist 
304 filelist

...around three hundred and four DNS-nameservice zonefiles in your cvs
working area that you really wanted to bulk-edit, but instead of their
being in a single directory, they're filed into a large directory tree
by first character, like this:

drwxrwxr-x  32 rmoen rmoen  4096 Feb 12 19:36 ./
drwxrwxr-x   7 rmoen rmoen  4096 Dec 18 18:32 ../
drwxrwxr-x   3 rmoen rmoen  4096 Oct 28 13:57 0/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:44 a/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:44 b/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:44 c/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:45 d/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:45 e/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:45 f/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:45 g/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:57 h/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:46 i/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 19:38 j/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 k/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 l/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:57 m/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 n/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 o/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:57 p/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 q/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 r/
drwxrwxr-x   3 rmoen rmoen 32768 Feb 12 18:25 reverse/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 s/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 t/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 u/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 v/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 w/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:26 x/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 y/
drwxrwxr-x   3 rmoen rmoen  4096 Feb 12 18:25 z/

Let's say, just to make life fun, that that tree is also littered with
CVS directories and other junk, which you want to carefully _not_ bulk-edit.

So, if you want to replace ns1.example.com throughout with ns7.example.com:

find . -type f | grep -v CVS | xargs -i sed -i 's/ns1.example.com/ns7.example.com/' {}

...and ns2.example.com with ns8.example.com:

find . -type f | grep -v CVS | xargs -i sed -i 's/ns2.example.com/ns8.example.com/' {}

...is one way to do that.  The "find . -type f" means "recursively find
all files of type regular (as opposed to directory, etc.) using current
dir = . as the starting point, and drilling down".  The "grep -v CVS"
says "but exclude from that list anything with 'CVS' in its name".

The "xargs -i" says "for each of the filenames thus generated, do a
command, with the ability to slot the filename into a specific place in
that command".

The "sed -i" says "The particular command I want you to do on each such
file is a sed = stream-edit operation, run in-line on the target file
directly.

The 's/ns1.example.com/ns7.example.com/' says "For each of those files, 
find all lines containing the expression "ns1.example.com", and
substitute "ns7.example.com" for that string.

The "{}" says "slot down the target filename exactly here, when you run
the command".


Oh, wait.  What's the one thing everyone forgets to do with zonefiles?
That's right, you always forget to advance the zonefile serial number,
and have to go back and edit it again.  Rather than do that 304 times, 
let's use our friends find, xargs, grep, and sed:

# Increment all S/Ns:
find . -type f | grep -v CVS | xargs -i sed -i 's/200[0-9]\{7\}/2009021300/g' {}


The sed search string this time, 200[0-9]\{7\}, is my attempt to
describe what a zonefile S/N is supposed to look like, which is basically 
YYYYMMDDnn (year, month, day, and two digits starting with "00", giving
you 100 available S/Ns each day).  And, of course, the right side,
"2009021300" is the first S/N for this day, Feb. 13, 2009.  (You have to
make very sure that S/Ns never go any direction but up, so I'd have to
be certain nobody else had done a zonefile update today.[1])


But wait!  There's more!  Now, you're ready to do cvs check-in.  And
it'd be no fun to run "cvs ci" 304 times either, so:

find . -type f | xargs -i egrep -l 'ns1.example.com|ns2.example.com' {} | xargs -i cvs ci -m "Updating NS lines for new nameservers" {}

"egrep" is somewhat smarter, "extended grep" (POSIX extended regular
expressions).  There are a number of differences that I'd rather not get
into, the one relevant here being a command syntax less likely to drive
you crazy.

The pipe "|" character is an "or" operator.  "cvs ci -m" means "cvs
checkin with a log message as follows".

Anyway, against expectation, it all actually works.


[1] Here's a shell script to actually _increment_ S/Ns, rather than just
assume today's "00" value is a good choice for absolute assingnment,
that one might use instead:


#!/bin/sh
file=$1

if [ ! -e "$file" ]; then
        echo "file not found, exiting"
        exit 1
fi

today=`date +%Y%m%d`
cur_serial=`head -n 5 $file | grep 200[456789] | awk '{print $1}'`
cur_date=`echo $cur_serial | cut -c 1-8`
cur_num=`echo $cur_serial | cut -c 9-10`

if [ "$today" -gt "$cur_date" ]; then
        new_date=$today
        new_num=00
elif [ "$today" = "$cur_date" ]; then
        new_date=$today
        new_num=`printf '%02d\n' $(expr $cur_num + 1)`
else
        echo "Error, current serial is in the future!  Aborting"
        exit 3
fi

new_serial=$new_date$new_num
echo "$file: Incremented serial from $cur_serial to $new_serial"