System News Bulletin
Wednesday, April 15, 2009
During severe power fluctuations indirectly caused by a wind storm,
a huge AC power spike or surge destroyed all hard drives, the motherboard,
all but one stick of RAM, and the power supply unit. Incidentally, the
spike or surge also triggered a 20 amp circuit breaker inline to the
server farm, but not in time to save the equipment. Substitute hardware
with an old backup came online a few hours later, with basic SMTP, Web,
and DNS service. Offsite backup data was retrieved over the following
two days. Services are being rebuilt using more-modern software that
is often subtly different from that on the dead machine, e.g., Apache httpd
2.x instead of 1.3.x, PHP5 instead of PHP4.
Done: Rebuild host, restore backup sets from offsite. Basic
SMTP, DNS, vsftpd, basic Apache2 service, CGI configs for PerlHoo and others,
mod_rewrite config, NTP sync, system crontabs, rsyncd,
PHPiCalendar / WebDAV, configure and lock down PHP5. Merge in post-backup
data from several sources (while taking care not to lose post-rebuild deltas).
Add back to the public file tree various file collections formerly removed
for lack of disk space. GNU Mailman and Exim4 configuration for mailing lists,
including virtual domain support. SMTP-time spamicity testing using
SA-Exim. Update BALE dataset.
To do: The list below will be updated
as they are brought back online:
- Anti-spam measures within the MTA (EximConfig base config set
including frontline Exim filtering rules, cached testing of RFC
compliance via MTA callouts, SMTP-time checking of SPF reference records)
- Local newsgroups (leafnode)
-- Rick Moen
Tuesday, February 1, 2005
The system was down for 22 hours for rebuild following Apache httpd
compromise. Debian-unstable's AWstats Web-statistics package turned
out to have had a serious unfixed bug whereby the "pluginmode" parameter
could be exploited in a call to the Perl routine eval(), allowing
attackers to execute arbitrary commands. For the near future at least,
we'll be regarding that thing as too buggy to run as a CGI, here. (Note to
sysadmins: You can run it as a cronjob that generates static pages
of Web statistics, instead of as a CGI — and should
think twice about making detailed system httpd stats publicly accessible,
Although there was almost certainly no host compromise, I rebuilt the
machine anyway, out of caution (which is what took 22 hours). It was
time for a redesign and rebuild, anyway.
I've also taken the occasion to eliminate several unwise PHP defaults
— that are appropriate for protected development servers, but not
ones deployed in public. The following php.ini environment variables
are now set to "Off" (as they should have been, long ago):
Ditto these, now changed from their "Off" defaults to "On":
...and, in line with the above, I uncommented "error_log=syslog"
(since debugging information should go there and not to the
Some of your PHP-based pages may need recoding, but (if you'll
pardon the expression) that's the breaks.
See also: http://linuxmafia.com/faq/Security/php.html for other aspects of PHP setup for the Web that should be carefully checked.
If you're a shell user, suddenly unable to SSH in, send me mail, or
telephone me at 650-561-9820. FYI: No files or mail were lost.
Tuesday, August 28, 2001
Three days ago, the system's 36GB hard drive catastrophically failed.
I've rebuilt the system on different hardware, and the really bad part
is having to revert to the May 9, 2001 backup: We've lost 3 1/2 months
worth of machine state (except for some pages for which Google had more
recent copies in cache). I will soon be contacting users to give everyone
Mailing lists/local newsgroups will be out of service until around a
week into September. Everything else should be back.
-- Rick Moen
Saturday, May 19, 2001
The system now appears stable, after two episodes of unwelcome excitement:
On Monday, April 16, 2001, Northpoint Communications cut off my connectivity,
when AT&T got around to dismembering the Northpoint NOC. Around five hours
later, I finished reconfiguring my household network to use substitute ADSL
service, on new IP addresses. Consequently, I also changed my DNS
nameservice to reflect the new IPs. (Some remote nameservers' caching
policies undoubtedly led them to ignore my nameservice change for up
to several additional days. If the nameserver you use does such
caching, that is your problem, not mine: Your failure to reach my
machine's current IP address doesn't mean it was "down".)
Tuesday, May 9, 2001, a lingering hardware problem flared up at the worst
possible time: The second hard drive, which has been becoming flaky,
refused access to the OS at the precise moment when I was installing new
core libraries, freezing the machine, rendering it unbootable, and making
backup and recovery difficult. (Another machine's emergency rebuild, just
prior, had destroyed my current backup.) Properly securing the current
files, deciding on a new software configuration, picking and supporting
a new filesystem (SGI's XFS), and rebuilding and restoring everything took
several days. Full service came back on-line Saturday morning, May 12.
The failing hard drive can now be retired, and I expect no further downtime.
-- Rick Moen
Friday, March 23, 2001
My main machine, linuxmafia.com AKA hugin.imat.com, may soon be off the Net
for up to three weeks, a period of outage that may start at any moment,
with no advance warning. My apologies for the inconvenience.
Essentially, I was caught napping by the imminent collapse of Northpoint
Communications, which has furnished SDSL transport to my house for my truly
superb bandwidth provider, VIA.NET. If you are a self-sufficient
computer user needing quality IP connectivity without the need to have
your hand held in the San Francisco Bay Area, you need look no further
than Joe McGuckin and his staff at VIA.NET.
Because of Northpoint's bankruptcy and dissolution, VIA.NET will probably
be unable to provide SDSL connectivity any more, and Northpoint's IP
connectivity may be shut off at any time. I have just placed an order for
replacement service through an equally highly reputable bandwidth provider
for self-sufficient users, Raw Bandwidth Communications, Inc. Unfortunately,
the lead time for provisioning is up to three weeks. (That delay is not
RBC's fault, but rather the local telco's.)
If/when my home connectivity is suddenly shut off, I may be able to
restore access by locating the machine elsewhere, in much less
than three weeks, but I cannot give advance details. There will also
be an inevitable lag, when that happens, for my new DNS information to
I will be able to be reached, during any lapse in service, at
firstname.lastname@example.org, or telephone
number 650-283-7902 (cellular).
-- Rick Moen