[sf-lug] improved: sf-lug.mbox ... rsync ... now gently backed up overnightly ...

Sat Mar 21 08:11:54 PDT 2015

Ah, bit behind my theoretical target timeline (was aiming to have this
done by first Saturday of this month, to get snagged by yet another set
of regular off-site backup rotations).

In any case ... now improved :-)
For a file that's append-mostly, why have lots of multiple copies (quite
redundant), and then at some point need to also have to be concerned
about rotating out older copies ... when there's better alternative.
Version control.  In this case, quite simple, since we're effectively
just dealing with different versions of one file, RCS suffices quite
well for that.  And taking a wee bit 'o care (using binary format, so no
unpleasant surprises with coincidental matching of RCS keywords or
uncompleted keyword strings, or other potential gotchas), have updated
the relevant code bits to store the earlier versions using RCS.
This also means, as least as far back as I (/we?) have it, older
versions of the sf-lug.mbox files are in RCS, and can be retrieved from:
http://www.sf-lug.org/RCS/sf-lug.mbox,v
And remains the case that most current version of sf-lug.mbox can be
retrieved from:
(upstream sources - either of these, the first preferable for updating)
rsync://linuxmafia.com/sf-lug/sf-lug.mbox
http://linuxmafia.com/pipermail/sf-lug.mbox/sf-lug.mbox
or nearly as current (updated daily downstream copy, presently higher
bandwidth but not made available via rsync):
http://www.sf-lug.org/sf-lug.mbox
Oh, and also, I'll soon be getting rid of (it goes bye-bye in about a
week):
http://www.sf-lug.org/sf-lug.mbox.BAK/

Details, references, etc.:
$ hostname && id
sflug
uid=25351(sflug) gid=25351(sflug) groups=25351(sflug)
$ (cd / && echo '
> >>/dev/null 2>&1 rm -rf /var/www/sf-lug/sf-lug.mbox.BAK; :' |
at now + 7 days)
warning: commands will be executed using /bin/sh
job 24 at Sat Mar 28 07:42:00 2015
$ ls -l /etc/cron.daily/local_sf-lug.mbox
-rwxr-xr-x 1 root root 93 Mar 21 07:07 /etc/cron.daily/local_sf-lug.mbox
$ cat /etc/cron.daily/local_sf-lug.mbox
#!/bin/sh
su - sflug -c 'sleep 600 && >>/dev/null 2>&1  
sf-lug.mbox_rsync+version_control; :'
$ type sf-lug.mbox_rsync+version_control
sf-lug.mbox_rsync+version_control is  
/home/sflug/bin/sf-lug.mbox_rsync+version_control
$ ls -d ~
/home/sflug
$ < ~/bin/sf-lug.mbox_rsync+version_control expand -t 4
#!/bin/sh

set -e # bail if we fail

# vi(1) :se tabstop=4

# update sf-lug.mbox via rsync, save changes using version_control

umask 022 # default to world readable, writable only by owner

# directory where we put the file when we're done
target_dir=/var/www/sf-lug

# options we want to use for our rsync operation:
rsync_opts='--quiet --checksum --times --sparse --partial  
--ignore-times --compress-level=9 --bwlimit=5'
# --compress-level=9 and --bwlimit=5 to go easy on bandwidth for upstream

# my private (not shared by anything else) temporary directory,
# use same filesytem as target_dir (so mv(1) can use rename(2))
my_tmp_dir="$target_dir"/.sf-lug.mbox.tmp

# (numeric) signals we'll trap on
trapsigs='1 2 3 15'

# throughout, we generally trap signals of interest to clean up and
# return appropriate exit value.
gotsig=
for sig in $trapsigs
do
     trap 'gotsig='"$sig" "$sig"
done

# if my target temporary directory already exists, remove it
rm -rf "$my_tmp_dir" || :

trap 'rm -rf "$my_tmp_dir"' 0
for sig in $trapsigs
do
     trap 'trap - 0; rm -rf "$my_tmp_dir"; exit '$(expr "$sig" + 128) "$sig"
done
[ x"$gotsig" = x ] || kill -"$gotsig" "$$"

# make my private temporary directory and cd into it
mkdir "$my_tmp_dir"
cd "$my_tmp_dir"

# this target will be the rsync target we update
cp -p "$target_dir"/sf-lug.mbox "$my_tmp_dir"/sf-lug.mbox

# in case rsync cares, set it writeable for user before rsync update
chmod u=rw,go=r sf-lug.mbox

# update target in temporary location
rsync $rsync_opts rsync://linuxmafia.com/sf-lug/sf-lug.mbox sf-lug.mbox

cd "$target_dir"

# check that existing file matches what was most recently checked in
co -p -q -kb sf-lug.mbox |
>> /dev/null 2>&1 cmp - sf-lug.mbox || \
{
     # not matched, therefore not checked in, we go ahead and check it in
     rcs -l sf-lug.mbox && # lock it before check in
     chmod u=rw,go=r sf-lug.mbox && # make user writable as if did co -l
     # was earlier initialized with rcs -i -kb, so we default to
     # binary for, e.g. our ci -u, ci -l, co, etc.
     ci -d -u -M -m'update' sf-lug.mbox && # check it in
     chmod a=r sf-lug.mbox && # set read only for all
     # should now match, assert it (set -e):
     co -p -q -kb sf-lug.mbox |
     >>/dev/null 2>&1 cmp - sf-lug.mbox
}

# set user writable, in case mv(1) cares:
chmod u=rw,go=r "$my_tmp_dir"/sf-lug.mbox "$target_dir"/sf-lug.mbox
mv -f "$my_tmp_dir"/sf-lug.mbox "$target_dir"/sf-lug.mbox
rmdir "$my_tmp_dir"

# no longer need to clean up my_tmp_dir
trap - 0
for sig in $trapsigs
do
     trap 'trap - 0; exit '$(expr "$sig" + 128) "$sig"
done
# we hope our version control is "smart enough" to deal appropriately
# with any signals it may receive or any action it didn't earlier
# complete

# check if rsynced file matches what was most recently checked in
if co -p -q -kb sf-lug.mbox | >>/dev/null 2>&1 cmp - sf-lug.mbox; then
     chmod a=r sf-lug.mbox # set read only for all
else
     # not matched, therefore not checked in, we go ahead and check it in
     rcs -l sf-lug.mbox && # lock it before check in
     chmod u=rw,go=r sf-lug.mbox && # make user writable as if did co -l
     # was earlier initialized with rcs -i -kb, so we default to
     # binary for, e.g. our ci -u, ci -l, co, etc.
     ci -d -u -M -m'update' sf-lug.mbox && # check it in
     chmod a=r sf-lug.mbox && # set read only for all
     # should now match, assert it (set -e):
     co -p -q -kb sf-lug.mbox |
     >>/dev/null 2>&1 cmp - sf-lug.mbox
fi

http://linuxmafia.com/pipermail/sf-lug/2015q1/010740.html
http://linuxmafia.com/pipermail/sf-lug/2015q1/010666.html
http://linuxmafia.com/pipermail/sf-lug/2007q4/003232.html
http://linuxmafia.com/pipermail/sf-lug/2007q4/003228.html

> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
> To: sf-lug <sf-lug at linuxmafia.com>
> Subject: Re: sf-lug.mbox ... rsync ... now gently backed up overnightly
> Date: Sun, 01 Feb 2015 11:23:38 -0800

> Rick - Thanks
>
> regarding:
>> Date: Sun, 1 Feb 2015 01:15:56 -0800
>> From: Rick Moen <rick at linuxmafia.com>
>> To: sf-lug <sf-lug at linuxmafia.com>
>> Subject: Re: [sf-lug] sf-lug.mbox ... wget ... rsync? :-)
>> Message-ID: <20150201091556.GU10984 at linuxmafia.com>
>>
>> Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
>>
>>> Is http://linuxmafia.com/pipermail/sf-lug.mbox/sf-lug.mbox
>>> also available via such rsync access (and if so could you let me/us
>>> know the rsync path to it), or if it isn't could you make it so
>>> and let us know?  Thanks for your consideration on this.
>>
>> OK, added.  Pretty good idea!
>>
>> $ rsync linuxmafia.com::
>> sf-lug          SF-LUG cumulative mbox
>> $
>> :r /etc/rsyncd.conf
>> read only = yes
>> [sf-lug]
>>        path = /var/lib/mailman/archives/private/sf-lug.mbox/
>>        comment = SF-LUG cumulative mbox
>>        list = yes
>
> see also:
> http://linuxmafia.com/pipermail/sf-lug/2015q1/010666.html
>
> And now gently rsync(1)ed overnightly.  Still have bit more rotation
> stuff to add, but it's off to a fair start (and manually tested for
> first run).
> And this host to which it's rsync(1)ed, is itself backed up to off-site
> on a regular basis approximately monthly.
>
> The rsync(1)ed target copy (and older copies) available at:
> http://www.sf-lug.org/sf-lug.mbox
> http://www.sf-lug.org/sf-lug.mbox.BAK/
>
> And yes, I thought it a good idea - after I'd done the very first
> initial wget to the sflug host, seemed to me rsync would be the way to
> go to maintain it - avoid the wget --continue hazards, and go easy on
> server bandwidth.  Was only today or yesterday (time, priorities ...
> besides, had already at least gotten one recent copy) I managed to check
> and notice you already had public rsync access to the host - so seemed
> natural to then ask about such access to the file (which is public
> anyway).
>
> Still have some more rotation/thinning to add to the basic script, but
> what it thus far has is covered a bit further below (stripped of comments
> and blank lines, for brevity).
>
> Below, whitespace within [] is generally one space and one tab.
> PS1='# ' PS2='> '
>
> # hostname
> sflug
> # id
> uid=0(root) gid=0(root) groups=0(root)
> # crontab -l
> 07 3 * * * /usr/local/bin/sf-lug.mbox_rsync+rotate
> # < /usr/local/bin/sf-lug.mbox_rsync+rotate expand -t 4 |
>> sed -ne '1p;2,${;/^[  ]*[^    #]/p;}'
> #!/bin/sh
> set -e
> umask 022
> backup_and_staging_dir=/var/www/sf-lug/sf-lug.mbox.BAK
> target_dir=/var/www/sf-lug/
> rsync_opts='--quiet --checksum --times --sparse --partial
> --ignore-times --compress-level=9 --bwlimit=5'
> cd "$backup_and_staging_dir"
> cp -p sf-lug.mbox sf-lug.mbox.tmp
> rsync $rsync_opts rsync://linuxmafia.com/sf-lug/sf-lug.mbox sf-lug.mbox.tmp
> mtime=$(stat -c '%Y' sf-lug.mbox)
> if [ -f sf-lug.mbox."$mtime" ]; then
>     cmp sf-lug.mbox sf-lug.mbox."$mtime" &&
>     rm sf-lug.mbox
> else
>     mv sf-lug.mbox sf-lug.mbox."$mtime"
> fi
> mv sf-lug.mbox.tmp sf-lug.mbox
> ln -f sf-lug.mbox "$target_dir"/sf-lug.mbox
> find sf-lug.mbox.* -type d -prune -o -type f -mtime +366 -exec rm \{\} \;
> #