[sf-lug] sf-lug - roster stats & automated backup

[Michael Paoli]

For certain definitions of "fairly soon" ;-)

Anyway, wee bit 'o sf-lug list (roster) stats - followed by backup bits:
$ hostname && id && sf-lug_roster_stats
sflug
uid=25351(sflug) gid=25351(sflug) groups=25351(sflug)
YYYY-MM-DD
2015-03-15 316
2015-03-06 318
2015-02-06 319
2015-02-04 318
2015-02-02 319
2015-01-22 328
$
That shows number of email addresses on the sf-lug list roster,
for the various dates shown.

Also, did complete quite automating the collection of the list
roster and backing it up.  On Rick's (the linuxmafia.com) end,
roster is generated, encrypted, and placed for available pickup.
I've now put in place crontab job which picks that up, saves it,
decrypts it, and checks changes into version control.  Also wrote
script to generate some stats from that version control data -
specifically dates and list membership count.

Details, details (and earlier references, etc.) ...

In much of the below, where two or more whitespaces are seen within
[  ] or [^  ], the actual whitespace is generally one single space
character, and one tab character.

Oh, and the fair bit of sleep(1) in the below, is to give
upstream a good running head start, yet otherwise match to
same timing - to snag copy rather freshly after it's created
(scheduling is highly similar on the upstream).

# hostname && id && cat /etc/cron.daily/local_sf-lug-roster
sflug
uid=0(root) gid=0(root) groups=0(root)
#!/bin/sh
su - sflug -c 'sleep 300 && >>/dev/null 2>&1  
sf-lug_roster_fetch+decrypt+version_control; :'
# su - sflug
$ PS1='$ '
$ echo $PATH
/home/sflug/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
$ expand -t 4 < bin/sf-lug_roster_fetch+decrypt+version_control
#!/bin/sh

# :se tabstop=4

# this script,
# sf-lug_roster_wget+decrypt+RCS
# is to fetch and decrypt our upstream data source,
# and to also save older decrypted versions via version control

set -e # bail if we fail

umask 022

# Configuration stuff:
# URL of encrypted membership roster:
fetch_url='http://linuxmafia.com/pipermail/sf-lug.mbox/sf-lug_roster.asc'
target_www_dir=/var/www/sf-lug
target_www_base_file=sf-lug_roster.asc
target_www_temp_base_file=.sf-lug_roster.asc.tmp
login=sflug
HOME=$(perl -e 'print((getpwnam(q('"$login"')))[7],"\n");')
[ -n "$HOME" ]
target_dir="$HOME"/sf-lug_roster
target_base_file=sf-lug_roster
target_temp_base_file=.sf-lug_roster.tmp

# the below bit is some more information on what upstream
# file://linuxmafia.com/etc/cron.daily/sf-lug-roster
# (on linuxmafia.com) prepares for us:

# #!/bin/sh
# #
# # sf-lug-roster:  Cron script to save out current Mailman mailing list
# # sf-lug's roster in a place SF-LUG officers can get it.
# #
# #               Written by Rick Moen (rick at linuxmafia.com)
# #               $Id: cron.weekly,v 1.00 2015-02-02 16:06:00 rick
#
# set -o errexit  #aka "set -e": exit if any line returns non-true value
# set -o nounset  #aka "set -u": exit upon finding an uninitialised variable
#
# /var/lib/mailman/bin/list_members -f sf-lug | \
# /usr/bin/gpg --armor --yes --batch --trust-model always --encrypt  
--recipient \
# 0x960C4BE648737D4287DC188FE8A55E60878BD8C0 > \
# /var/lib/mailman/archives/private/sf-lug.mbox/sf-lug_roster.asc

# 06:32:04 typical daily local mtime of the above

# (numeric) signals we'll trap on
trapsigs='1 2 3 15'

# throughout, we generally trap signals of interest to clean up and
# return appropriate exit value.
gotsig=
for sig in $trapsigs
do
     trap 'gotsig='"$sig" "$sig"
done

temp_file=$(mktemp)

trap 'rm "$temp_file"' 0
for sig in $trapsigs
do
     trap 'trap - 0; rm "$temp_file"; exit '$(expr "$sig" + 128) "$sig"
done
[ x"$gotsig" = x ] || kill -"$gotsig" "$$"

# fetch file from our URL
wget -q -O "$temp_file" "$fetch_url"

for sig in $trapsigs
do
     trap 'trap - 0; rm -f "$temp_file"  
"$target_www_dir/$target_www_temp_base_file"; exit '$(expr "$sig" +  
128) "$sig"
done

# to yet another temporary location (on same filesystem),
# we do this as the first is or may be faster and more efficient and
# automatically cleaned up periodically and/or upon reboots,
# and the latter, as it's on same filesystem as target, so mv(1) can use
# rename(2)
cp -p "$temp_file" "$target_www_dir/$target_www_temp_base_file"
rm "$temp_file"

trap 'trap - 0; rm -f "$target_www_dir/$target_www_temp_base_file"; exit 0' 0
for sig in $trapsigs
do
     trap 'trap - 0; rm -f  
"$target_www_dir/$target_www_temp_base_file"; exit '$(expr "$sig" +  
128) "$sig"
done

# all good so far, move to its persistent location
mv -f "$target_www_dir/$target_www_temp_base_file"  
"$target_www_dir/$target_www_base_file"

umask 077 # roster contents confidential (strongly encrypted is public)

trap 'rm "$target_dir/$target_temp_base_file"' 0
for sig in $trapsigs
do
     trap 'trap - 0; rm "$target_dir/$target_temp_base_file"; exit  
'$(expr "$sig" + 128) "$sig"
done

# decrypt
< "$target_www_dir/$target_www_base_file" \
> "$target_dir/$target_temp_base_file" \
gpg --decrypt

# take mtime from source
touch -m -r "$target_www_dir/$target_www_base_file" \
"$target_dir/$target_temp_base_file"

# is file contents same as before?
if >>/dev/null 2>&1 cmp "$target_dir/$target_temp_base_file" \
     "$target_dir/$target_base_file"; then
     # same contents, we're esentially done
     exit 0 # trap removes temporary file
else
     # updated contents - put in place, update version control
     mv -f "$target_dir/$target_temp_base_file" \
     "$target_dir/$target_base_file"
     trap - 0
     for sig in $trapsigs
     do
         trap 'trap - 0; exit '$(expr "$sig" + 128) "$sig"
     done
     cd "$target_dir"
     rcs -l "$target_base_file"
     ci -d -u -M -m'update' "$target_base_file"
fi
$

And for stats, we have:
$ expand -t 4 < bin/sf-lug_roster_stats
#!/bin/sh
cd /home/sflug/sf-lug_roster &&
rlog sf-lug_roster |
sed -ne '
     /^revision ./{
         s/^revision //
         h
     }
     /^date: [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9]  
[0-9][0-9]:[0-9][0-9]:[0-9][0-9];/{
         s/^date:  
\([0-9][0-9][0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9]\)  
[0-9][0-9]:[0-9][0-9]:[0-9][0-9];.*$/\1-\2-\3/
         H
         s/.*//
         x
         s/\n/ /
         /^[^    ][^     ]* [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$/p
         d
     }
' |
{
     echo 'YYYY-MM-DD'
     while read r yyyymmdd
     do
         echo "$yyyymmdd" \
         $(2>>/dev/null co -p"$r" sf-lug_roster | wc -l)
     done
}
$

> From: "Michael Paoli" <Michael.Paoli at cal.berkeley.edu>
> To: "Rick Moen" <rick at linuxmafia.com>
> Cc: sf-lug <sf-lug at linuxmafia.com>
> Subject: Thanks!: Re: [sf-lug] ... list_members -f sf-lug | ...
> Date: Mon, 02 Feb 2015 16:28:25 -0800

> Excellent, thanks, works perfectly fine.  I'll have it automated on the
> receiving end fairly soon.
>
>> From: "Rick Moen" <rick at linuxmafia.com>
>> Subject: Re: [sf-lug] ... list_members -f sf-lug | ...
>> Date: Mon, 2 Feb 2015 15:13:38 -0800
>>
>> Quoting Michael Paoli (Michael.Paoli at cal.berkeley.edu):
>>
>>> If you've got that, you can use gpg to decrypt the bit
>>> that's encrypted further below - it's simple symmetric encryption using
>>> that same password as passphrase for the encryption.
>>
>> Fingerprint matches what you sent in unencrypted SMTP, which I Totally
>> and Completely Trust.  (A joke.  Yes, we used a shared secret, so we're
>> good.)
>
> Yes, shared secret - low tech start, but can also be usefully utilized
> to effectively bootstrap other authentication/security (e.g. when one of
> two users hasn't already generated their PGP key for keysigning event,
> or the highly common case, a user's initial temporary password).
>
>> Manually running the export:
>>
>> # /var/lib/mailman/bin/list_members -f sf-lug | gpg --armor --yes  
>> --batch --trust-model always --encrypt --recipient  
>> 0x960C4BE648737D4287DC188FE8A55E60878BD8C0 >  
>> /var/lib/mailman/archives/private/sf-lug.mbox/sf-lug_roster.asc
>> #
>>
>> Seems to have worked.  This cron rendition appears to produce the
>> expected output, too:
>>
>> :r /etc/cron.daily/sf-lug-roster
>>
>> #!/bin/sh
>> #
>> # sf-lug-roster:  Cron script to save out current Mailman mailing list
>> # sf-lug's roster in a place SF-LUG officers can get it.
>> #
>> #               Written by Rick Moen (rick at linuxmafia.com)
>> #               $Id: cron.weekly,v 1.00 2015-02-02 16:06:00 rick
>>
>> set -o errexit  #aka "set -e": exit if any line returns non-true value
>> set -o nounset  #aka "set -u": exit upon finding an uninitialised variable
>>
>> /var/lib/mailman/bin/list_members -f sf-lug | \
>> /usr/bin/gpg --armor --yes --batch --trust-model always --encrypt  
>> --recipient \
>> 0x960C4BE648737D4287DC188FE8A55E60878BD8C0 > \
>> /var/lib/mailman/archives/private/sf-lug.mbox/sf-lug_roster.asc
>> That should do it.  You be the judge.  ;->  Accordingly, I'm removing
>> your e-mail as target for older cron job /etc/cron.weekly/mailman-rosters .
>>
>
> Thanks, and yes, tested, looks perfectly fine!  :-)
> I was able to snag the file via http and decrypt it without problem.