Alasdair on Everything   

Boot environments with OpenSolaris domUs on xVM

August 23rd, 2010

I was having a nightmare with an OpenSolaris domU xen guest I had broken. It had multiple boot environments, all I needed was to boot one of them – the default rpool/ROOT/opensolaris one would do. However when booting the guest, no grub menu was presented by pygrub, it just goes straight to booting the kernel:

root ~ (xvm03.everycity): xm start guest -c
v3.4.2-xvm chgset 'Wed Feb 17 02:28:56 2010 -0800 19674:ba65a89eded4'
SunOS Release 5.11 Version snv_147 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
/platform/i86xpv/kernel/drv/amd64/rootnex: undefined symbol 'iommulib_nexdma_unmapobject'
WARNING: mod_load: cannot load module 'rootnex'

panic[cpu0]/thread=fffffffffbc609e0: BAD TRAP: type=e (#pf Page fault) rp=fffffffffbcb4ec
 addr=20 occurred in module "genunix" due to a NULL pointer dereference
...
skipping system dump - no dump device configured
rebooting...

I spent ages trying to get the grub menu to display, but couldn’t work out how to, so decided to take a different approach.

First, I needed a list of boot environments out of the guest. If you already know which boot environment you want to boot (such as the original rpool/ROOT/opensolaris boot environment) then you can skip this bit. I include it as it’s quite a nifty method – thanks to a friend from IRC, he provided me with this clever script:

#!/usr/bin/python2.4
import sys
import pygrub
import fsimage

disk = sys.argv[1]
file = sys.argv[2]

fs = fsimage.open(disk, pygrub.get_fs_offset(disk))
f = fs.open_file(file)

print f.read();

Pop this into a file called readfile.py. This script steals some libraries from pygrub, so you’ll need to copy this to the same working directory. The usage is as follows:

# cp /usr/lib/xen/bin/pygrub ./pygrub.py
# ./readfile.py /dev/zvol/dsk/rpool/xvm/guest /boot/grub/menu.lst

This will then spit back the grub menu list at you:

# python2.4 readfile.py /dev/zvol/dsk/data/xvm/guest/disk0 /boot/grub/menu.lst
timeout 5
default 3
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title OpenSolaris Development snv_134
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/opensolaris
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive
...

From this list, the information we want is the “bootfs” argument from the boot environment you want to boot. So for example rpool/ROOT/opensolaris. We can then shove this into the xvm xml config file for the guest:

<domain type='xen'>
  <name>guest</name
  <memory>4194304</memory>
  <currentMemory>4194304</currentMemory>
  <vcpu>8</vcpu>
  <bootloader>/usr/lib/xen/bin/pygrub</bootloader>
 <bootloader_args>--args="-B zfs-bootfs=rpool/ROOT/opensolaris"</bootloader_args> 
  <os>
    <type>linux</type>
  </os>
...

The key line above there is the bootloader_args argument – this passes the correct bootfs to the kernel telling it which ZFS filesystem to use for the boot. Hey presto, your guest should now boot :)

Entry Filed under: General