[conspire] Other notes from the Debian 5.0.1/Lenny to 6.0/Squeeze upgrade

Rick Moen rick at linuxmafia.com
Tue Aug 24 16:27:52 PDT 2010

So, in the process of forcing my server to lurchingly migrate to current
Debian-testing, which the 'Squeeze' branch that has been assigned
version 6.0 in anticipation of release, lots of unexpected breakage
occurred.  One thing I've noticed is that, when doing a massive apt-get
upgrade operation, it's really vital to _copy and paste_ into a buffer
anything from the package advisory notices that might be important.

Here's a bunch of that, and comments:

   Consider installing grub-legacy.

   In order to replace the Legacy version of GRUB in your system, it is
   recommended that /boot/grub/menu.lst is adjusted to chainload GRUB 2
   from your existing GRUB Legacy setup.  This step may be automaticaly
   performed now.             

   It's recommended that you accept chainloading GRUB 2 from menu.lst,  and
   verify that your new GRUB 2 setup is functional for you, before you
   install it directly to your MBR (Master Boot Record).             

   In either case, whenever you want GRUB 2 to be loaded directly from MBR,
   you can do so by issuing (as root) the following command:


Oh joy.  Yet another bootloader.  I might just put LILO back.

For now, we'll see if GRUB2 does what it's supposed to.  I have not yet
done a reboot following the upgrade adventure.

   Unable to migrate to dependency-based boot system

   Tests have determined that problems in the boot system exist which
   prevent migration to dependency-based boot sequencing:

   package bittorrent left obsolete init.d script behind, package discover1
   left obsolete init.d script behind, package makedev left obsolete init.d  
   script behind, insserv: warning: script 'K20vsftpd' missing LSB tags and
   overrides, insserv: warning: script 'S25libdevmapper1.02' missing LSB
   tags and overrides, insserv: warning: script 'libdevmapper1.02' missing  
   LSB tags and overrides, insserv: script setserial: service setserial 
   already provided!, insserv: warning: script 'vsftpd' missing LSB tags
   and overrides,
   If the reported problem is a local modification, it needs to be fixed 
   manually. If it's a bug in the package, it should be reported to the BTS
   and fixed in the package. See
   http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot for more 
   information about how to fix the problems preventing migration.

   To reattempt the migration process after the problems have been fixed,
   run "dpkg-reconfigure sysv-rc".           

Seems like the scripts griping that my startup scripts cannot be
optimised to ensure that everything automagically starts in the correct 
order.  I can live with that.

   (Reading database ... 49019 files and directories currently installed.)
   Preparing to replace udev 0.114-2 (using .../archives/udev_160-1_i386.deb) ...

   Since release 150, udev requires that support for the CONFIG_SYSFS_DEPRECATED
   feature is disabled in the running kernel.

   Please upgrade your kernel before or while upgrading udev.

   AT YOUR OWN RISK, you can force the installation of this version of udev
   AT THE NEXT REBOOT by creating the /etc/udev/kernel-upgrade file.
   There is always a safer way to upgrade, do not try this unless you
   understand what you are doing!

   dpkg: error processing /var/cache/apt/archives/udev_160-1_i386.deb
   subprocess new pre-installation script returned error exit status 1

That's a serious problem, and dpkg did the right thing under present 
circumstances by _refusing_ to install a udev package that would 
break the entire /dev device-file handling for the currently running
2.6.26-2-686 binary kernel package.  When I get around to rebooting,
I'll make a point of upgrading the udev package -then- but not before.

Current udev subsystem is working just peachy, and I'm sure not going to
touch it.

Omitted from literal display, but will be discussed:  apt-get offering
me a choice of what to do about replacements for the following files:

   vs. /etc/spamassassin/local.cf.dpkg-new 
   vs. /etc/exim4/conf.d/main/02_exim4-config_options.dpkg-new
   vs. /etc/ntp.conf.dpkg-new

This is, again, a serious matter that should _not_ be gone through
lightly.  If you're careless, you can lose vital local configuration
state and revert to generic defaults.  You get a prompt like this one:

  Setting up exim4-config (4.72-1) ...

  Configuration file `/etc/exim4/conf.d/main/02_exim4-config_options'
   ==> Modified (by you or by a script) since installation.
   ==> Package distributor has shipped an updated version.
     What would you like to do about it ?  Your options are:
      Y or I  : install the package maintainer's version
      N or O  : keep your currently-installed version
        D     : show the differences between the versions
        Z     : start a shell to examine the situation

I _usually_ do the "D" (which pauses installation and shows you a diff,
taking you back to the same question prompt when you leave the diff), 
and then either do "Y" or "N" depending.  If I say "N", then I make a
point of reviewing the *.dpkg-new file ASAP, because there are always 
new things that deserve going over.

MySQL nightmare:

Upon trying to start the new MySQL 5.1 daemon manually (or at least, I
_think_ this is how I found the core of the problem):

   Fatal error: Can't open and lock privilege tables: Incorrect key file
   for table 'host'; try to repair it

(Or, I might have found that in a logfile.  Anyway, it was the nub of
the problem, as it turned out.)

Part of the package-installation message stream shown as the debacle

   Aborting downgrade from (at least) 5.1 to 5.0.

   Errors were encountered while processing:

   dpkg: dependency problems prevent configuration of mysql-server:
   mysql-server depends on mysql-server-5.1; however:
     Package mysql-server-5.1 is not configured yet.
   dpkg: error processing mysql-server (--configure):
    dependency problems - leaving unconfigured
   Errors were encountered while processing:
   Created commit 91e47aa: committing changes after apt run
   Setting up mysql-server-5.1 (5.1.49-1) ...
   Stopping MySQL database server: mysqld.
   Starting MySQL database server: mysqld . . . . . . . . . . . . . .  failed!
   invoke-rc.d: initscript mysql, action "start" failed.
   dpkg: error processing mysql-server-5.1 (--configure):
   subprocess installed post-installation script returned error exit status 1
   dpkg: dependency problems prevent configuration of mysql-server:
    mysql-server depends on mysql-server-5.1; however:
     Package mysql-server-5.1 is not configured yet.
   dpkg: error processing mysql-server (--configure):
   dependency problems - leaving unconfigured
  Errors were encountered while processing:

So, MySQL cannot pass dkpg installation because one of the steps, that
of starting MySQL daemon at the end of installing the package, has
failed for an unspecified reason.  Examination reveals that the daemon
insists the database files are damaged, and that you should run
utilities to repair them.

However, the repair utilities require that mysqld be running.  Which
can't run because it doesn't like the database files.  So, I went on a
laborious quest to manually remove all the 5.1 package plus any packages
that depend on them.  I also had to realise that I had to remove
null-length file /var/lib/mysql/debian-5.1.flag -- or else apt-get/dpkg
would balk and issue that 'Aborting downgrade from (at least) 5.1 to
5.0' message.

Finally, a full set of MySQL 5.0 packages -- the version I'd been
running -- were back.  I attempted database repair:  The repair
utilities claimed nothing needed fixing.

Just because I figured would _probably_ be needed as a brute-force way
of discarding and then recreating database files that MySQL 5.1 would
like, I did this as root:  
   mysqldump --password=[password] --all-databases > alltables-$(date +%F).sql

That gave me an alltables-2010-08-23.sql file that had everything from
the cumulative database, as pure SQL.

Now, re-upgrade to 5.1.  As I feared, 5.1 still rejected the database
files.  Stop mysqld.  Move the database files in question from
/var/lib/mysql to /var/lib/mysql-2010-08-23.  Start mysqld.  Now, mysqld
creates blank, empty database files that it likes.  Now do:

   mysql  --password=[password] < alltables-$(date +%F).sql

And the SQL that created the database contents in the first place 
gets replayed as a SQL sequence, reloading all data.

(Oddly, for some reason, the definition of MySQL user 'bale' and 
its table rights was not recreated, so the BALE and CABAL pages' event
listings were broken until I manually recreated that user.)

    These devices will be assigned UUIDs or labels:

    /dev/sdc8: UUID=46b8d092-234f-4241-969b-a54fff38445d
    /dev/sda5: UUID=dded258a-f6ce-4a4c-9ee7-5d8076648080

    These configuration files will be updated:
    /etc/fstab, /boot/grub/menu.lst, /etc/initramfs-tools/conf.d/resume  

    The device IDs will be changed as follows:
    /dev/sda6: UUID=728b737e-8420-437c-a43a-5d5a8f60fba5
    /dev/sdc8: UUID=46b8d092-234f-4241-969b-a54fff38445d
    /dev/sda9: UUID=5f702fae-9386-436e-86d2-90323b7f0857
    /dev/sda7: UUID=0dc3dcff-3d63-4830-893f-6f9afd811875
    /dev/sda8: UUID=ed5d4608-6db8-40c0-9405-ba091b5f8a77
    /dev/sda1: UUID=96ce6da4-7b55-409e-a078-3572612b61c1
    /dev/sdb8: UUID=336b0268-44ba-4302-834f-d51b55e53a6b
    /dev/sda5: UUID=dded258a-f6ce-4a4c-9ee7-5d8076648080

    Apply configuration changes to disk device IDs?

I really loathe UUIDs, but I'm going to let it tinker with my files at
least long enough to make sure everything still mounts at next reboot.

   Configuration files still contain deprecated device names
  The following configuration files still use some device names that may
  change when using the new kernel:
  /etc/fstab: /dev/hda 

That's the ATAPI ('IDE') CD-ROM drive.  Which could change to /dev/sdc0
at next reboot.  Whatever.  I'll figure it out then.

  Required firmware files may be missing

  This system is currently running Linux 2.6.26-2-686 and you are
  installing Linux 2.6.32-5-686.  In the new version some of the drivers 
  used on this system may require additional firmware files:

   e100: e100/d102e_ucode.bin, e100/d101s_ucode.bin, e100/d101m_ucode.bin  

   Most firmware files are not included in the system because they do not 
   conform to the Debian Free Software Guidelines. You may need to
   reconfigure the package manager to include the contrib and non-free
   sections of the package archive before you can install these firmware 


  Running update-initramfs.
  update-initramfs: Generating /boot/initrd.img-2.6.32-5-686
  W: Possible missing firmware /lib/firmware/e100/d102e_ucode.bin for module e100
  W: Possible missing firmware /lib/firmware/e100/d101s_ucode.bin for module e100
  W: Possible missing firmware /lib/firmware/e100/d101m_ucode.bin for module e100

Whoa, that is _very_ serious, and you can bet I jumped on that, right
quick.  The 'e100' is the Intel ethernet chipset inside the VA Linux
Systems model 2230's Intel L440GX+ "Lancewood" Pentium III motherboard.
If the kernel used at next boot lacks the firmware-image files for it, 
then my server would be unable to talk to the network, and I'd need to
carry the fixes to it via a USB flash drive or something.

After some futile searching around, e.g.

   apt-file search /lib/firmware/e100/d102e_ucode.bin

...I just found and installed new packages firmware-linux-free and
firmware-linux-nonfree, then re-ran 'update-initramfs'.  It now was able
to build the initial RAMdisk filesystem, needed for booting the new
kernel, without errors.

Also very useful was the fact that I have logcheck installed, which
sends the administrator sundry grumbles parsed from /var/log/messages.
This turned up, among other things, advance warnings of breakage that
will occur in versions of PHP5 yet to come.  E.g., there were warnings
that a couple of lines in php.ini adjusted 'deprecated' PHP functions.
I disabled them after I hunted them down in /etc/php5/apache2/php.ini:

; Deprecated starting PHP 5.3, hence disabled 2010-08-24.
; register_long_arrays = On

; Deprecated starting PHP 5.3, hence disabled 2010-08-24.
; magic_quotes_gpc = On

Simiarly, the PHP5 interpreter grumbled that a couple of PHP modules
could not be loaded (on account of unfixed bugs to be fixed post 5.3).
They turned out to be functions I have no use for, so I just removed the 
packages, which I believe were php5-gd and php5-mcrypt.

Also, important:  Before engaging in any such adventure, make a safety
reference copy of the contents of /etc.  I do it like this:

   tar cvzf /root/etc-$(date +%F).tar.gz /etc

It also doesn't hurt to save an ASCII copy of your current package

  dpkg --get-selections "*" > /root/selections-$(date +%F)

And snapshots of your partition maps:

  fdisk -l /dev/sda > /root/partitions-sda-$(date +%F)
  fdisk -l /dev/sdc > /root/partitions-sdc-$(date +%F)

If worst comes to worst and you need to rebuild your system, that
captures the essence of your system other than your data files (which
you have backed up, right)?

More information about the conspire mailing list