System News Bulletin




Wednesday, April 15, 2009

During severe power fluctuations indirectly caused by a wind storm, a huge AC power spike or surge destroyed all hard drives, the motherboard, all but one stick of RAM, and the power supply unit. Incidentally, the spike or surge also triggered a 20 amp circuit breaker inline to the server farm, but not in time to save the equipment. Substitute hardware with an old backup came online a few hours later, with basic SMTP, Web, and DNS service. Offsite backup data was retrieved over the following two days. Services are being rebuilt using more-modern software that is often subtly different from that on the dead machine, e.g., Apache httpd 2.x instead of 1.3.x, PHP5 instead of PHP4.

Done: Rebuild host, restore backup sets from offsite. Basic SMTP, DNS, vsftpd, basic Apache2 service, CGI configs for PerlHoo and others, mod_rewrite config, NTP sync, system crontabs, rsyncd, PHPiCalendar / WebDAV, configure and lock down PHP5. Merge in post-backup data from several sources (while taking care not to lose post-rebuild deltas). Add back to the public file tree various file collections formerly removed for lack of disk space. GNU Mailman and Exim4 configuration for mailing lists, including virtual domain support. SMTP-time spamicity testing using SA-Exim. Update BALE dataset.

To do: The list below will be updated as they are brought back online:

  • Anti-spam measures within the MTA (EximConfig base config set including frontline Exim filtering rules, cached testing of RFC compliance via MTA callouts, SMTP-time checking of SPF reference records)
  • Local newsgroups (leafnode)

-- Rick Moen







Tuesday, February 1, 2005

The system was down for 22 hours for rebuild following Apache httpd compromise. Debian-unstable's AWstats Web-statistics package turned out to have had a serious unfixed bug whereby the "pluginmode" parameter could be exploited in a call to the Perl routine eval(), allowing attackers to execute arbitrary commands. For the near future at least, we'll be regarding that thing as too buggy to run as a CGI, here. (Note to sysadmins: You can run it as a cronjob that generates static pages of Web statistics, instead of as a CGI — and should think twice about making detailed system httpd stats publicly accessible, anyway.)

Although there was almost certainly no host compromise, I rebuilt the machine anyway, out of caution (which is what took 22 hours). It was time for a redesign and rebuild, anyway.

I've also taken the occasion to eliminate several unwise PHP defaults — that are appropriate for protected development servers, but not ones deployed in public. The following php.ini environment variables are now set to "Off" (as they should have been, long ago):

  • register_globals
  • allow_url_fopen
  • file_uploads
  • y2k_compliance
  • allow_call_time_pass_reference
  • expose_php
  • display_errors
  • html_errors

Ditto these, now changed from their "Off" defaults to "On":

  • zlib.output_compresson
  • log_errors

...and, in line with the above, I uncommented "error_log=syslog" (since debugging information should go there and not to the public Web).

Some of your PHP-based pages may need recoding, but (if you'll pardon the expression) that's the breaks.

See also: http://linuxmafia.com/faq/Security/php.html for other aspects of PHP setup for the Web that should be carefully checked.

If you're a shell user, suddenly unable to SSH in, send me mail, or telephone me at 650-561-9820. FYI: No files or mail were lost.

--Rick Moen







Tuesday, August 28, 2001

Three days ago, the system's 36GB hard drive catastrophically failed. I've rebuilt the system on different hardware, and the really bad part is having to revert to the May 9, 2001 backup: We've lost 3 1/2 months worth of machine state (except for some pages for which Google had more recent copies in cache). I will soon be contacting users to give everyone fresh passwords.

Mailing lists/local newsgroups will be out of service until around a week into September. Everything else should be back.

-- Rick Moen






Saturday, May 19, 2001

The system now appears stable, after two episodes of unwelcome excitement:

On Monday, April 16, 2001, Northpoint Communications cut off my connectivity, when AT&T got around to dismembering the Northpoint NOC. Around five hours later, I finished reconfiguring my household network to use substitute ADSL service, on new IP addresses. Consequently, I also changed my DNS nameservice to reflect the new IPs. (Some remote nameservers' caching policies undoubtedly led them to ignore my nameservice change for up to several additional days. If the nameserver you use does such caching, that is your problem, not mine: Your failure to reach my machine's current IP address doesn't mean it was "down".)

Tuesday, May 9, 2001, a lingering hardware problem flared up at the worst possible time: The second hard drive, which has been becoming flaky, refused access to the OS at the precise moment when I was installing new core libraries, freezing the machine, rendering it unbootable, and making backup and recovery difficult. (Another machine's emergency rebuild, just prior, had destroyed my current backup.) Properly securing the current files, deciding on a new software configuration, picking and supporting a new filesystem (SGI's XFS), and rebuilding and restoring everything took several days. Full service came back on-line Saturday morning, May 12.

The failing hard drive can now be retired, and I expect no further downtime.

-- Rick Moen






Friday, March 23, 2001

Dear Folks:

My main machine, linuxmafia.com AKA hugin.imat.com, may soon be off the Net for up to three weeks, a period of outage that may start at any moment, with no advance warning. My apologies for the inconvenience.

Essentially, I was caught napping by the imminent collapse of Northpoint Communications, which has furnished SDSL transport to my house for my truly superb bandwidth provider, VIA.NET. If you are a self-sufficient computer user needing quality IP connectivity without the need to have your hand held in the San Francisco Bay Area, you need look no further than Joe McGuckin and his staff at VIA.NET.

Because of Northpoint's bankruptcy and dissolution, VIA.NET will probably be unable to provide SDSL connectivity any more, and Northpoint's IP connectivity may be shut off at any time. I have just placed an order for replacement service through an equally highly reputable bandwidth provider for self-sufficient users, Raw Bandwidth Communications, Inc. Unfortunately, the lead time for provisioning is up to three weeks. (That delay is not RBC's fault, but rather the local telco's.)

If/when my home connectivity is suddenly shut off, I may be able to restore access by locating the machine elsewhere, in much less than three weeks, but I cannot give advance details. There will also be an inevitable lag, when that happens, for my new DNS information to propagate.

I will be able to be reached, during any lapse in service, at rick@deirdre.net, or telephone number 650-283-7902 (cellular).

-- Rick Moen