squeeze upgrade checklist

Riseup squeeze upgrade checklist

This is riseup’s lenny to squeeze upgrade checklist. Most of it is pretty generic, things that are Riseup specific are marked with RISEUP. Things that apply to physical hosts, that do not apply to virtual hosts, are marked with REAL.

Preparation

  • make sure you have backups
  • make sure lenny packages are up to date so you are starting from a known point
  • REAL: make sure the remote console works before starting
  • run screen, you dont want to get disconnected
  • record the session for later: script -t 2>~/upgrade-squeezestep.time -a ~/upgrade-squeezestep.script
  • disable puppet
  • RISEUP: switch node in site.pp to use squeeze and any other needed changes
  • review any pending actions in aptitude (In some cases, the use of apt-get for installing packages instead of aptitude might make aptitude consider a package as “unused” and schedule it for removal. In general, you should make sure the system is fully up-to-date and “clean” before proceeding with the upgrade.) launch aptitude in “visual mode” and press g (“Go”). If it shows any actions, you should review them and either fix them or implement the suggested actions. If no actions are suggested you will be presented with a message saying “No packages are scheduled to be installed, removed, or upgraded” (section 4.2.1 of release notes)
  • dpkg --audit

Cleanup

  • check for packages installed but that aren’t in an an archive we’re using. apt-show-versions |grep -v uptodate. Review the list and then here’s a gun (don’t shoot yourself) apt-show-versions |grep -v uptodate|awk '{print $1}' |grep -v syslog-ng |xargs apt-get --purge -y remove
  • apt-get autoremove
  • purge packages that are removed but still in rc state. Here’s another footgun for name in `dpkg -l |grep ^rc|awk '{print $2}'`;do dpkg -P $name;done

Upgrading

  • remove (or edit) /etc/apt/preferences to make sure squeeze is not excluded
  • edit /etc/apt/sources.list to point to squeeze (remove volatile if you are using it)
  • apt-get update
  • make sure enough space: apt-get -o APT::Get::Trivial-Only=true dist-upgrade
  • minimal upgrade first:
    • apt-get -d upgrade first to download and check that it looks ok
    • announce you are stopping services and upgrading the system
    • if on a machine with complicated services that might break as different components are upgraded (examples: a mail server that also depends on remote database, a list server that needs to talk to smtp and database) then potentially remove that service from use (change DNS, etc) and stop the services to ensure no data loss
    • apt-get upgrade Here are the RISEUP answers to some questions you might see
      • freetype for rendering
      • /etc/login.defs changes are ok, hit Y
      • The /etc/pam.d/login changes will get fixed by puppet, hit Y
      • restarting things for pam is ok
      • dbconfig-common changes are ok, install the new
      • /etc/mysql/my.cnf will get overwritten by puppet, keep the existing
      • if there are sysctl.conf changes review them and see if they are needed
      • don’t take the backupninja.conf changes, they will get overwritten
      • /etc/apticron/apticron.conf will get overwritten by puppet, keep the existing
  • REAL: apt-get install linux-image-2.6-flavor ; apt-get install udev
    • when asked if you want the new device names for PATA devices, review what it will do and make sure it’s right
  • compete the upgrade
    • apt-get -du dist-upgrade and review what it’s going to do, and if ok accept to download all the packages first
    • apt-get -u dist-upgrade to actually start the upgrade. Here are the RISEUP answers to some questions you might see
      • REAL: yes chainload from menu.lst for grub-pc
      • yes we want dash to be /bin/sh
      • ack the tmpreaper thing
      • restarting things for libc6 is ok
      • The /etc/bash.bashrc will get overwritten anyway, accept N
      • libpam-runtime, unix auth profile is ok
      • yes we want dependency-based booting
      • The /etc/pam.d/cron changes will get fixed by puppet, hit Y
      • The /etc/pam.d/atd changes will get fixed by puppet, hit Y
      • The /etc/init.d/postfix changes are ok, hit Y (but if you are upgrading something that processes mail this means postfix will restart and you may want to login and stop it)
      • The /etc/bind/db.root changes will get overwritten anyway, accept N
      • We don’t want the /etc/bind/named.conf change, accept N
      • The /etc/default/shorewall changes will get overwritten, accept N for both
      • The syslog-ng changes will get overwritten, accept N
      • The /etc/apticron/apticron.conf changes will get overwritten, pick “keep the local”
      • We don’t want the /etc/stunnel/stunnel.conf or /etc/default/stunnel4 changes, accept N
      • if the machine has a mysql server, you will need to pick certain additions and changes to the my.cnf and update the site-mysql/files/$fqdn/my.cnf and push that so puppet can pick it up. There is at least one option from lenny that will keep the DB from starting (skip-bdb)
  • upgrade completed
  • REAL: make sure we have the correct linux-2.6-* metapackage: dpkg -l "linux-image*" | grep ^ii
  • REAL: if on an machine that requires it (for example an lp1000r), you will need to install firmware-linux-nonfree to get the firmware
  • RISEUP: reenabled and run puppet until things are good
  • repeat the Cleanup section above to clean up anything that got created by the upgrade
    • if you got complaints about setting up dependency based booting, hopefully it will work now that we’ve done some clean up, run dpkg-reconfigure sysv-rc
    • RISEUP: take syslog-ng off hold so puppet won’t complain every run echo syslog-ng install | dpkg --set-selections
  • REAL: make sure /etc/fstab, /etc/crypttab are using UUIDs
  • REAL: check /etc/default/grub, make sure the following are set, and run update-grub2 if change any:
    GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="serial console"
#GRUB_GFXMODE=640x480
  • REAL: check the initrd to make sure its good:
    # mkdir /root/initrd ; cd /root/initrd
# zcat /boot/initrd.img-2.6.32-5-vserver-686-bigmem | cpio --extract
# cd conf/conf.d (look around make sure stuff is right)
  • exit the script session to save your upgrade log
  • exit your screen session
  • reboot
  • REAL: try to boot into grub2, make sure it works (and serial works), if it does, run this after bootup:
    upgrade-from-grub-legacy
  • re-add the service back into production (DNS, etc)
  • announce the service downtime and upgrade are complete