[sf-lug] Import of offlist messages: seems to have worked

[Rick Moen]

If you have a look at http://linuxmafia.com/pipermail/sf-lug/, it seems
like the mbox that Bobbie put together with help from Daniel 'imported'
cleanly.  You guys made it simple for me.  Please spot-check the 
archive, but at a quick glance it looks OK.

For the record, here's what happened.




1.  We started with a 54MB mbox file, which was the existing
/var/lib/mailman/archives/private/sf-lug.mbox/sf-lug.mbox:

-rw-r--r-- 1 list list 55589790 Mar  1 12:28 sf-lug.mbox

Bobbie and Daniel provided these additional 2MB of messages, that 
Daniel and Bobbie imported from Bobbie's POP3 access and imported into
Mozilla Thunderbird (which like many traditionalist e-mail programs 
uses mbox format for its main storage):

-rw------- 1 danielg4 danielg4 1765942 Mar  1 12:03 interim.mbox


2.  I just concatenated the two, then moved the result into place:


cd /var/lib/mailman/archives/private/sf-lug.mbox
cp sf-lug.mbox ~rick  #for backup purposes
cat sf-lug.mbox /tm/nterim.mbox  >  sf-lug.temp
mv sf-lug.tmp sf-lug.mbox
chown list:list sf-lug.mbox  #Set file ownership correctly.

3.  Rebuild:

Kind of caveman-like so far, right?  Just glued those suckers together.  
Now, the only halfway clever part:  Mailman's 'pipermail' utility is a
Python program that parses mbox files and constructs HTML and ASCII
archives from them.  This _can_ go badly wrong if pipermail fails to 
parse the file correctly, e.g., you can end up with lots of fragmentary
pieces of messages presented as junk in the online archive.  But the
nice thing is that, having saved a copy of the previous mbox, if
necessary I can revert.

The pipermail utility is bin/arch within Mailman's /var/lib/mailman
base directory, so I went there, and issude the command to wipe the
entire existing archive and rebuild the whole thing, adding the '-q'
(quiet) flag to suppress progress output.  With a 56MB file, rebuilding
the archive takes about 10 minutes.

cd ../../..   #This takes us to Mailman's basedir, /var/lib/mailman .
bin/arch --wipe -q sf-lug archives/private/sf-lug.mbox/sf-lug.mbox

Wait ten minutes, you get your prompt back, you're done.  

(I actually erred and carried out these actions as the root user, 
where I should have done them as 'list', the Mailman user, but caught
my error, went back, and corrected some file ownerships.)


Quick-reference help for the pipermail (bin/arch) utility:

inuxmafia:/var/lib/mailman# bin/arch
Rebuild a list's archive.

Use this command to rebuild the archives for a mailing list.  You may want to
do this if you edit some messages in an archive, or remove some messages from
an archive.

Usage: bin/arch [options] <listname> [<mbox>]

Where options are:
    -h / --help
        Print this help message and exit.

    -q / --quiet
        Make the archiver output less verbose.

    --wipe
        First wipe out the original archive before regenerating.  You usually
        want to specify this argument unless you're generating the archive in
        chunks.

    -s N
    --start=N
        Start indexing at article N, where article 0 is the first in the mbox.
        Defaults to 0.

    -e M
    --end=M
        End indexing at article M.  This script is not very efficient with
        respect to memory management, and for large archives, it may not be
        possible to index the mbox entirely.  For that reason, you can specify
        the start and end article numbers.

Where <mbox> is the path to a list's complete mbox archive.  Usually this will
be some path in the archives/private directory.  For example:

% bin/arch mylist archives/private/mylist.mbox/mylist.mbox

<mbox> is optional.  If it is missing, it is calculated.

listname is required
linuxmafia:/var/lib/mailman#