Mailing lists using usenet newsgroup

[Rick Moen]

On Tue, Jan 6, 2015 at 5:54 PM, jim <jim at well.com> wrote:

> As usual, you are entirely correct, if overly polite.

On the contrary, Jim, you're a generous and kind-hearted person who
deserves every respect and courtesy.

I often tell people, functional, usable backups are the most important
thing in computing.  Hardware you can replace by buying it.  Software
you can reinstall.  Software configurations you can re-create, costing
you only time and trouble.  But data... when your data are lost,
that's gone forever.

When I had a WAN/LAN consulting business in the 1990s, my first
regular customer was a clothing manufacturing firm in SOMA named NE
Wear.  (They don't exist any more - not my fault.)  The first few
visits, I was solving problems with their small-scale LAN and server
setup.  They relied mostly on a NetWare server in the main management
office, and there was among other things a DAT DDS2 tape drive and a
reasonable rotational backup scheme using some enterprise backup
program like Arcserve.  I was reviewing everything; they had the right
number of tapes, etc.  (DDS2 tapes' big advantage was high capacity at
very low cost per GB, and also speed.)

Suddenly I boggled about something I was not seeing, like Conan
Doyle's Sherlock Holmes story where the gimmick is Sherlock noticing a
dog who didn't bark in the night.  I got the office manager's
attention, pointed to the tape drive, and said 'Excuse me, but where
are the cleaning cartridges?'  And she said the Words of Ill Omen,
'What's a cleaning cartridge?'  As I feared, whoever set them up had
never covered this detail, nor ever scheduled 'restore' sessions to
verify recoverability.  This was A Big Problem, because DAT tape
drives use helical-scan heads, a technology first applied to
videotapes.  In helical-scan recording, as the tape glides past the
heads, the heads, which are round in cross-section and mounted on a
spoke, are also spinning vertically rapidly, in contact with the tape
that's passing by them.  Thus, the head-contact pattern is helical;
thus the name.  Helical-scan heads have advantages for recording
density but abrade the passing tape rapidly, wearing iron-oxide
recording substrate off the tape.  The iron oxide abraded from the
tape transfers to the heads.  As you use additional tapes in the
drive, the abraded material speads around from worn, dirty tapes via
the heads to clean, newer tapes, gradually getting spread onto all the
tapes and progressively ruining them.  The only way to slow this cycle
of degradation is... cleaning cartridges, which polish the surface of
the magnetic heads.  Each pass of a cleaning cartridge wears the head,
and eventually you must send your drive out to get new heads, so you
don't overuse cleaning cartridges - but under no circumstances do you
do without them.

Which brings me badk to NE Wear.  Almost certainly, all of their
existing backup tapes were 100% unreadable.  And they didn't have any
idea, until I noticed the missing cleaning cartridges.

I discussed the solution with the office manager, I bought some
supplies, and then we held a low-key but important meeting with the
office staff.  I gave the nickel tour of the problem, and pointed out
three things on the table.

First thing was a pair of cleaning cartridges.  These were to be used
on some schedule going forward.  (This might have been an item in the
backup software; can't remember.)  I stressed that I'd just used one
of them five times in a row, to get the heads clean, and that such a
drastic cleaning was necessary but unfortunate, because it would have
worn a lot of surface from the heads.  We wanted to (please) not
create the problem situation again.

Second thing was a brand new supply of unopened DDS2 tapes.  I said,
important:  You must use ONLY these tapes for backup from this time
forward, because all the old tapes have loose iron oxide coating
smeared all over them and would, if used, smear that debris onto the
heads again, recreating the problem.

Third thing was a cardboard box, into which I'd put all of the old
tapes, and atop the lid was a note saying 'Do not use without
explicitl permission of $office-manager or consultant Rick Moen.'  We
would keep the old tapes ONLY in case of a dire emergency need to
restore old data, which I estimated would be unsuccessful.

The manager and I stressed that, if the above guidelines weren't
followed, that we'd be back to square one and having this meeting all
over again. adding polluted new tapes to the bad-tapes box, and
running the cleaning cartridge five times again, losing more head
surface.

A couple of weeks later, I found that someone was being dumb and the
old tapes were in circulation again, so we went back to square one,
retired polluted new tapes, abraded more surface off the tape heads,
and held another meeting.

Second time was the charm.

I had two take-aways from the experience.  One was the epiphany that
the firm's continued existence had been in my hands.  If I hadn't
noticed the missing cleaning cartridge (or alternatively attempted
test restores and found the backup sets unusable), the firm could have
easily lost all of their electronic business records in a hard drive
failure - while tragically assuming they were covered because of
backups.  And only my sharp-eyed attention had saved them.

The other was a running gag I used from that point forward, with all
my consulting clients.  That was to refer to the backup system, tongue
in cheek, as a 'restore system', the point being that any backup
system that doesn't restore is worse than pointless (because of the
false sense of security), and that restoring was the operation of
utmost importance, hence the operation it should be named for.

Anyway:  Backups, backups, backups.  Restores, restores, restores.

>
>
> Consider yourself "part of" for sure.
> * the mailing list issue strikes me as deserving
>    further conversation and some patience. Let's
>    continue general discussion, for sure, but with
>    no expectation of immediate action.
> * an archive for sf-lug and other lugs and such        <----
>    strikes me as a great idea that might be quickly   <----
>    adopted. Discussion could begin with the merits   <----
>    of Digital Ocean and comparison with other          <----
>    more home-brewed efforts (e.g. using a host        <----
>    that belongs to one of us, is internet-accessible,   <----
>    and allows multiple administrators. <----
> * please suggest any other projects and ideas that
>    we might share.
>
>
>
>
> On 01/06/2015 04:21 PM, LEdWorldwide!> wrote:
>>
>> All of this talk of mailing lists sounds like too much fun to pass up the
>> opportunity to be a part of (and it gives me a chance to put my LPI
>> certification to good use).
>>
>> I'm enthusiastic about taking on the project and can get the data from
>> Rick.  I also like the idea of using Mailman and I'd like to consider
>> Digital Ocean to host a VPS for us ($5-$10 per month).
>>
>> I'm certainly open to other ideas/suggestions but I do know that as a
>> group we can pool our resources and knowledge together and create some great
>> projects.
>>
>>
>> Cheers,
>>
>> -Michael Rojas-
>>
>>
>> Rick Moen <rick at deirdre.net> wrote:
>>
>>> On Tue, Jan 6, 2015 at 2:40 PM, jim <jim at well.com> wrote:
>>>>
>>>> I'm for MailMan. Either we wait until Rick gets to
>>>> it or we set up our own (after we get the backups
>>>> from Rick).
>>>
>>> I'm pretty sure I've sent this information to you-plural in the past,
>>> but I'll do it now just in case:   This is intended to help your
>>> collective memory and improve your process going forward.
>>>
>>> 1.  GNU Mailman can be trivially configured (by the site admin) to
>>> permit public download of the cumulative mbox of any hosted Maliman
>>> mailing list.  That is the set of all past postings to date, from
>>> which the archives can then be recreated anywhere desired, with a
>>> single command (using /var/lib/mailman/bin/arch).  Thus, it is 95% of
>>> the important data comprising the mailing list's 'state'.  In the case
>>> of SF-LUG's mailing list on linuxmafia.com, the relevant URL was (and
>>> will again be) /http://linuxmafia.com/pipermail/sf-lug.mbox/sf-lug.mbox
>>> .  Not all Mailman instances are configured to enable that function,
>>> but all of the ones that I administer are.
>>>
>>> If I never brought that matter to Jim's attention (and I'm pretty sure
>>> I did), I'm doing so now.  Yr. welcome.
>>>
>>> It is common sense for you to periodically back up that file, and
>>> doing so is greatly in your interest.  Shortly before I started
>>> hosting a mailing list for SF-LUG, you guys completely lost all back
>>> traffic to a previous iteration of the SF-LUG mailing list somewhere
>>> else -- to something like a failed hard drive or such -- and at the
>>> time I wondered why you never bothered to back up the mbox, given your
>>> then-recent somewhat ignominious loss.  E.g., anyone whatsoever could
>>> do that task daily or weekly using a simple 'wget -c' fetch in a cron
>>> job.
>>>
>>> 2.  The other indispensable part of a Mailman mailing list's state is
>>> the subscriber roster.  The mailing list admin can arrange to have
>>> that information mailed to interested parties periodically using the
>>> /var/lib/mailman/bin/list_members utility.   We might call that 4% of
>>> the mailing list's 'state'.  The other 1% would be things like any
>>> unusual mailing list settings, individual subscriber's subscription
>>> passwords, and the like, none of which is a huge loss if you happen to
>>> lose it.
>>>
>>> In the case of my server, all mailing list information is present on
>>> both my backups and on the live hard drives of the (down) server.  If
>>> you-all decide you wish to get the current data, you can visit my
>>> house and bring a Linux machine able to read an ext3 filesystem from a
>>> USB device.  I can easily give you the cumulative mbox file during
>>> your visit.  For the roster, I can give you Mailman's stored copy,
>>> which is in a Python's 'pickle' stored-data format.  You would need to
>>> figure out how to extract what you need from that.
>>>
>>> You have my cellular number.
>>>
>>> Might I suggest that you guys start showing some basic initiative
>>> towards self-preservation?  If you had bothered to do that, you would
>>> not have ignominiously lost your entire previous hard drive, and you
>>> would not be needing to ask me for 'backups from Rick'.
>
>