How to do a full backup and restore of linuxmafia.com
(Example cheatsheet with casual, full backup of a Debian GNU/Linux system.)

Backup — execute these tasks:

fdisk -l /dev/sda > /root/partitions-sda-$(date +%F)  Partition map of SCSI HD a
fdisk -l /dev/sdb > /root/partitions-sdb-$(date +%F)  Partition map of SCSI HD b
fdisk -l /dev/sdc > /root/partitions-sdc-$(date +%F)  Partition map of SCSI HD c
dpkg --get-selections "*" > /root/selections-$(date +%F)  Installed packages
tar cvzf /root/etc-$(date +%F).tar.gz /etc

Then, back up these directories and files, e.g., over rsync to a remote system for safekeeping. 

/root                        Root user's home directory (includes above files)
/etc                         System configuration files
/usr/lib/cgi-bin             CGI scripts (omit PHP binaries)
/var/lib/mysql               MySQL database files (dump if not queiscent)
/boot/grub/menu.lst          GRUB bootloader configuration
/var/spool/exim4             Exim and SA-Exim internal files
/var/spool/news              NNTP news spool for Leafnode
/var/mail                    SMTP mail spool
/var/lib/mailman/archives    Mailing list archives for Mailman
/var/lib/mailman/data        Mailing list state and other data
/var/lib/mailman/lists       Mailing list definitions for Mailman
/var/lib/mailman/nntp        Mailing list NNTP gateway data
/var/lib/mailman/qfiles      Mailing list in-process data
/usr/local                   Locally installed files and records
/var/www                     Public http, ftp, rsync tree
/home                        Non-root users' home trees

Part of the point of the above is that including all of the directories that need backup is a non-trivial task, and requires some study of one's system. (Backing up everything comprising a Linux system, as often suggested, would be monumentally dumb, a huge waste of backup space and backup/restore time. E.g., you'll want any rebuild to use current distro packaged software freshly installed from trusted sources, not the old installed binaries copied back.)

Here is the current, dirt-simple script. People often advise me it could be so very much better, which is probably true, but the point is it is simple, transparent, and verifiably works well, and also that it's actually here: One of my mottos is that the backup/restore methods you actually have and have tested beat the theoretically better ones you haven't gotten around to implementing, every single time.

This script appears to capture all of that except /boot/grub/menu.lst:

#!/bin/sh
cd /backup
rsync -axP --delete /var/lib/ var-lib ;\
rsync -axP --delete /var/www/ var-www ;\
rsync -axP --delete /home/ home ;\
rsync -axP --delete /var/spool/ var-spool ;\ 
rsync -axP --delete /var/mail/ var-mail ;\
rsync -axP --delete /usr/local/ usr-local ;\
rsync -axP --delete /root/ root ;\
rsync -axP --delete /usr/lib/cgi-bin/ usr-lib-cgi-bin
cd / ; tar czf etc-$(date +%F).tar.gz /etc

Dump MySQL tables:

mysqldump -uroot -pSECRETWORD --all-databases > alltables-$(date +%F).sql

Copy all of the above, plus /boot/grub/menu.lst to offline archival storage.

Networked equivalent:

#!/bin/sh
cd /backup
ssh root@linuxmafia.com cd / ; tar czf - /etc > etc-$(date +%F).tar.gz
ssh root@linuxmafia.com mysqldump -uroot -pSECRETWORD --all-databases > alltables-$(date +%F).sql
rsync -axP --delete root@linuxmafia.com:/var/lib/ var-lib
rsync -axP --delete root@linuxmafia.com:/var/www/ var-www
rsync -axP --delete root@linuxmafia.com:/home/ home
rsync -axP --delete root@linuxmafia.com:/var/spool/ var-spool
rsync -axP --delete root@linuxmafia.com:/var/mail/ var-mail
rsync -axP --delete root@linuxmafia.com:/usr/local/ usr-local
rsync -axP --delete root@linuxmafia.com:/root/ root
rsync -axP --delete root@linuxmafia.com:/usr/lib/cgi-bin/ usr-lib-cgi-bin

Restore:

If the above-cited system needed to be reloaded from backup, here is how it would be restored:

Do a minimal Debian installation from trusted media, using partition maps preserved above. 

mkdir /usr/local/opt
rmdir /opt && ln -s /usr/local/opt /opt

Prune the contents of (a copy of) file selections-NNNN-NN-NN from above to remove (1) libs (which will get fetched if needed as dependencies and (2) packages you're pretty sure upon examination you no longer need, then, do 

 dpkg --set-selections < selections-[date string]
 ## Many inconsequential warnings about "package not in database".
 apt-get update
 apt-get install dselect
 dselect update
 apt-get dselect-upgrade

This may necessitate some interative pruning and correcting of the "selections-NNNN-NN-NN" file, as it's quite possible (especially if you're using a later Debian base version as the migration target) that some formerly used packages no longer exist / have different names. Don't worry too much if you can't find a package's new name; more likely than not, it will be found as a dependency anyway. However, make a note of anything that appears unfindable, for later checking.

Rebuild /etc with reference to /etc copy above, making sure you restore users and groups to /etc/passwd, shadow, group, gshadow. For linuxmafia.com, build/configure leafnode, configure Mailman's nntp service, enable related cronjobs.

Restore MySQL tables from dump file. 

mysql -u root -p
mysql> create database events;
mysql> use events;
mysql> source alltables-NNNN-NN-NN.sql;

It will also be desirable to set MySQL's password for root and the user that BALE runs as, e.g.: 

mysql -u root mysql
mysql> UPDATE mysql.user SET Password = PASSWORD('SECRETWORD') WHERE User = 'root'; FLUSH PRIVILEGES;
mysql>

2006-05 linuxmafia.com network services:

Copy other files back.

Rebuild or reinstall local packages.

2006-05 local packages on linuxmafia.com:

Full packages:

One-off scripts and utilities:

Restart services. Check logfiles for errors.