Mailing lists using usenet newsgroup

[jim]

Thank you.

     Your story reminds me of my SA days at the USCA9.
We did backups every night.
     We did not have tape head cleaners nor did we play
back tapes to verify restore.

     I believe that you several times wrote about backing
up the MailMan data and meta-data, yet I was possibly
* too unfamiliar with some aspects of Unix and *nix systems
* too distracted or stressed or lazy to explore
* needed more involvement with others sharing the same
sense of immediacy
     I'm not trying to defend or excuse myself; rather I
hope my confession helps someone reflect and possibly
improve.

     I hope we stick with you (involvement, immediacy,
expertise...) running MailMan for other-than-LinuxMafia
groups.
     Assuming so, seems to me that sf-lug and dv-lug might
benefit with a shared backup regimen. My idea is to find
out what MailMan facilities allow full backup as well as
incremental backup:
* somebody (hopefully not me, but I'm willing) catches the
    appropriate MailMan data to their laptop system, verifies
    the data restore, then makes sure the data is on some
    redundant storage device (my vote is USB stick or SSD
    or some such).
* verification, seems to me, will require a MailMan
    installation that sufficiently matches that which is on the
    LinuxMafia host. Installing such a MailMan system
    strikes me as a good exercise for those interested in
    developing skills and having fun.



On 01/07/2015 06:52 AM, Rick Moen wrote:
> On Tue, Jan 6, 2015 at 5:54 PM, jim <jim at well.com> wrote:
>
>> As usual, you are entirely correct, if overly polite.
> On the contrary, Jim, you're a generous and kind-hearted person who
> deserves every respect and courtesy.
>
> I often tell people, functional, usable backups are the most important
> thing in computing.  Hardware you can replace by buying it.  Software
> you can reinstall.  Software configurations you can re-create, costing
> you only time and trouble.  But data... when your data are lost,
> that's gone forever.
>
> When I had a WAN/LAN consulting business in the 1990s, my first
> regular customer was a clothing manufacturing firm in SOMA named NE
> Wear.  (They don't exist any more - not my fault.)  The first few
> visits, I was solving problems with their small-scale LAN and server
> setup.  They relied mostly on a NetWare server in the main management
> office, and there was among other things a DAT DDS2 tape drive and a
> reasonable rotational backup scheme using some enterprise backup
> program like Arcserve.  I was reviewing everything; they had the right
> number of tapes, etc.  (DDS2 tapes' big advantage was high capacity at
> very low cost per GB, and also speed.)
>
> Suddenly I boggled about something I was not seeing, like Conan
> Doyle's Sherlock Holmes story where the gimmick is Sherlock noticing a
> dog who didn't bark in the night.  I got the office manager's
> attention, pointed to the tape drive, and said 'Excuse me, but where
> are the cleaning cartridges?'  And she said the Words of Ill Omen,
> 'What's a cleaning cartridge?'  As I feared, whoever set them up had
> never covered this detail, nor ever scheduled 'restore' sessions to
> verify recoverability.  This was A Big Problem, because DAT tape
> drives use helical-scan heads, a technology first applied to
> videotapes.  In helical-scan recording, as the tape glides past the
> heads, the heads, which are round in cross-section and mounted on a
> spoke, are also spinning vertically rapidly, in contact with the tape
> that's passing by them.  Thus, the head-contact pattern is helical;
> thus the name.  Helical-scan heads have advantages for recording
> density but abrade the passing tape rapidly, wearing iron-oxide
> recording substrate off the tape.  The iron oxide abraded from the
> tape transfers to the heads.  As you use additional tapes in the
> drive, the abraded material speads around from worn, dirty tapes via
> the heads to clean, newer tapes, gradually getting spread onto all the
> tapes and progressively ruining them.  The only way to slow this cycle
> of degradation is... cleaning cartridges, which polish the surface of
> the magnetic heads.  Each pass of a cleaning cartridge wears the head,
> and eventually you must send your drive out to get new heads, so you
> don't overuse cleaning cartridges - but under no circumstances do you
> do without them.
>
> Which brings me badk to NE Wear.  Almost certainly, all of their
> existing backup tapes were 100% unreadable.  And they didn't have any
> idea, until I noticed the missing cleaning cartridges.
>
> I discussed the solution with the office manager, I bought some
> supplies, and then we held a low-key but important meeting with the
> office staff.  I gave the nickel tour of the problem, and pointed out
> three things on the table.
>
> First thing was a pair of cleaning cartridges.  These were to be used
> on some schedule going forward.  (This might have been an item in the
> backup software; can't remember.)  I stressed that I'd just used one
> of them five times in a row, to get the heads clean, and that such a
> drastic cleaning was necessary but unfortunate, because it would have
> worn a lot of surface from the heads.  We wanted to (please) not
> create the problem situation again.
>
> Second thing was a brand new supply of unopened DDS2 tapes.  I said,
> important:  You must use ONLY these tapes for backup from this time
> forward, because all the old tapes have loose iron oxide coating
> smeared all over them and would, if used, smear that debris onto the
> heads again, recreating the problem.
>
> Third thing was a cardboard box, into which I'd put all of the old
> tapes, and atop the lid was a note saying 'Do not use without
> explicitl permission of $office-manager or consultant Rick Moen.'  We
> would keep the old tapes ONLY in case of a dire emergency need to
> restore old data, which I estimated would be unsuccessful.
>
> The manager and I stressed that, if the above guidelines weren't
> followed, that we'd be back to square one and having this meeting all
> over again. adding polluted new tapes to the bad-tapes box, and
> running the cleaning cartridge five times again, losing more head
> surface.
>
> A couple of weeks later, I found that someone was being dumb and the
> old tapes were in circulation again, so we went back to square one,
> retired polluted new tapes, abraded more surface off the tape heads,
> and held another meeting.
>
> Second time was the charm.
>
> I had two take-aways from the experience.  One was the epiphany that
> the firm's continued existence had been in my hands.  If I hadn't
> noticed the missing cleaning cartridge (or alternatively attempted
> test restores and found the backup sets unusable), the firm could have
> easily lost all of their electronic business records in a hard drive
> failure - while tragically assuming they were covered because of
> backups.  And only my sharp-eyed attention had saved them.
>
> The other was a running gag I used from that point forward, with all
> my consulting clients.  That was to refer to the backup system, tongue
> in cheek, as a 'restore system', the point being that any backup
> system that doesn't restore is worse than pointless (because of the
> false sense of security), and that restoring was the operation of
> utmost importance, hence the operation it should be named for.
>
> Anyway:  Backups, backups, backups.  Restores, restores, restores.
>
>>
>> Consider yourself "part of" for sure.
>> * the mailing list issue strikes me as deserving
>>     further conversation and some patience. Let's
>>     continue general discussion, for sure, but with
>>     no expectation of immediate action.
>> * an archive for sf-lug and other lugs and such        <----
>>     strikes me as a great idea that might be quickly   <----
>>     adopted. Discussion could begin with the merits   <----
>>     of Digital Ocean and comparison with other          <----
>>     more home-brewed efforts (e.g. using a host        <----
>>     that belongs to one of us, is internet-accessible,   <----
>>     and allows multiple administrators. <----
>> * please suggest any other projects and ideas that
>>     we might share.
>>
>>
>>
>>
>> On 01/06/2015 04:21 PM, LEdWorldwide!> wrote:
>>> All of this talk of mailing lists sounds like too much fun to pass up the
>>> opportunity to be a part of (and it gives me a chance to put my LPI
>>> certification to good use).
>>>
>>> I'm enthusiastic about taking on the project and can get the data from
>>> Rick.  I also like the idea of using Mailman and I'd like to consider
>>> Digital Ocean to host a VPS for us ($5-$10 per month).
>>>
>>> I'm certainly open to other ideas/suggestions but I do know that as a
>>> group we can pool our resources and knowledge together and create some great
>>> projects.
>>>
>>>
>>> Cheers,
>>>
>>> -Michael Rojas-
>>>
>>>
>>> Rick Moen <rick at deirdre.net> wrote:
>>>
>>>> On Tue, Jan 6, 2015 at 2:40 PM, jim <jim at well.com> wrote:
>>>>> I'm for MailMan. Either we wait until Rick gets to
>>>>> it or we set up our own (after we get the backups
>>>>> from Rick).
>>>> I'm pretty sure I've sent this information to you-plural in the past,
>>>> but I'll do it now just in case:   This is intended to help your
>>>> collective memory and improve your process going forward.
>>>>
>>>> 1.  GNU Mailman can be trivially configured (by the site admin) to
>>>> permit public download of the cumulative mbox of any hosted Maliman
>>>> mailing list.  That is the set of all past postings to date, from
>>>> which the archives can then be recreated anywhere desired, with a
>>>> single command (using /var/lib/mailman/bin/arch).  Thus, it is 95% of
>>>> the important data comprising the mailing list's 'state'.  In the case
>>>> of SF-LUG's mailing list on linuxmafia.com, the relevant URL was (and
>>>> will again be) /http://linuxmafia.com/pipermail/sf-lug.mbox/sf-lug.mbox
>>>> .  Not all Mailman instances are configured to enable that function,
>>>> but all of the ones that I administer are.
>>>>
>>>> If I never brought that matter to Jim's attention (and I'm pretty sure
>>>> I did), I'm doing so now.  Yr. welcome.
>>>>
>>>> It is common sense for you to periodically back up that file, and
>>>> doing so is greatly in your interest.  Shortly before I started
>>>> hosting a mailing list for SF-LUG, you guys completely lost all back
>>>> traffic to a previous iteration of the SF-LUG mailing list somewhere
>>>> else -- to something like a failed hard drive or such -- and at the
>>>> time I wondered why you never bothered to back up the mbox, given your
>>>> then-recent somewhat ignominious loss.  E.g., anyone whatsoever could
>>>> do that task daily or weekly using a simple 'wget -c' fetch in a cron
>>>> job.
>>>>
>>>> 2.  The other indispensable part of a Mailman mailing list's state is
>>>> the subscriber roster.  The mailing list admin can arrange to have
>>>> that information mailed to interested parties periodically using the
>>>> /var/lib/mailman/bin/list_members utility.   We might call that 4% of
>>>> the mailing list's 'state'.  The other 1% would be things like any
>>>> unusual mailing list settings, individual subscriber's subscription
>>>> passwords, and the like, none of which is a huge loss if you happen to
>>>> lose it.
>>>>
>>>> In the case of my server, all mailing list information is present on
>>>> both my backups and on the live hard drives of the (down) server.  If
>>>> you-all decide you wish to get the current data, you can visit my
>>>> house and bring a Linux machine able to read an ext3 filesystem from a
>>>> USB device.  I can easily give you the cumulative mbox file during
>>>> your visit.  For the roster, I can give you Mailman's stored copy,
>>>> which is in a Python's 'pickle' stored-data format.  You would need to
>>>> figure out how to extract what you need from that.
>>>>
>>>> You have my cellular number.
>>>>
>>>> Might I suggest that you guys start showing some basic initiative
>>>> towards self-preservation?  If you had bothered to do that, you would
>>>> not have ignominiously lost your entire previous hard drive, and you
>>>> would not be needing to ask me for 'backups from Rick'.
>>