On a fresh Ubuntu install with Xen (learning Xen, proficient with Ubuntu), I try to create a domU PV VM with LUKS encryption, the whole setup procedure goes smoothly, including creating, unlocking the encrypted volumes and completing the full installation.
On reboot, however, it fails back to initramfs.
I would appreciate any help to get me on the right track to fixing this set-up, as I feel it must be pretty close, something silly may be in the way here that the right knowledge can unlock.
My set-up:
HW: 2011 i7 8GB RAM vt-x + vt-d, new SSD
Host OS (dom0): Ubuntu 18.04, no encryption
Xen: 4.9
Guest OS (domU #1): Ubuntu 18.04
Disk volumes in dom0 for domU
(dom0) /dev/vg0/lv-1 20GB setup as /dev/xvda (domU)
(dom0) /dev/vg0/lv-1swap 12GB setup as /dev/xvdb (domU)
Disk volumes on domU:
/dev/xvda partitioned as:
/dev/xvda1 ext4 1GB for /boot (bootable, noatime)
/dev/xvda5 available space, LUKS volume
/dev/xvdb1 available space, LUKS volume
/dev/xvda5_crypt ext4 / (noatime)
/dev/xvdb1_crypt swap
Starting domU 1 for the first time after install, console output (using screen):
# xl create -c /etc/xen/domu1.cfg
Parsing config from /etc/xen/domu1.cfg
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg
WARNING:root:grub2's saved_entry/next_entry not supported
WARNING:root:Unknown directive else
WARNING:root:Unknown directive menuentry_id_option
WARNING:root:Unknown directive else
WARNING:root:Unknown directive menuentry_id_option
WARNING:root:Unknown directive export
WARNING:root:Unknown directive font
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive font
WARNING:root:Unknown directive load_video
WARNING:root:Unknown directive terminal_output
WARNING:root:Unknown directive else
WARNING:root:Unknown directive elif
WARNING:root:Unknown directive clear
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive export
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video
WARNING:root:Unknown image directive gfxmode
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video
WARNING:root:Unknown image directive gfxmode
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive knetbsd
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown directive source
WARNING:root:Unknown directive elif
WARNING:root:Unknown directive source
pyGRUB version 0.6
┌────────────────────────────────────────────────────────────────────────┐
│ Ubuntu │
│ Ubuntu, with Linux 4.15.0-36-generic │
│ Ubuntu, with Linux 4.15.0-36-generic (recovery mode) │
│ Memory test (memtest86+) │
│ Memory test (memtest86+, serial console 115200) │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Use the ^ and ┴ keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.
[ 0.000000] e820: Cannot find an available gap in the 32-bit address range
[ 0.000000] e820: PCI devices with unassigned 32-bit BARs may not work!
[ 0.452369] dmi: Firmware registration failed.
Please unlock disk xvda5_crypt**********
cryptsetup (xvda5_crypt): set up successfully
Please unlock disk xvdb1_crypt**********
cryptsetup (xvdb1_crypt): set up successfully
/dev/xvda1: clean, 311/61056 files, 27841/243968 blocks
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /run on /root/run failed: No such file or directory
run-init: opening console: No such file or directory
Target filesystem doesn't have requested /sbin/init.
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
No init found. Try passing init= bootarg.
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
There seems to be something wrong out of the box with my grub configuration, however it still offers the boot options, proceeds to boot, requests the LUKS passwords, but then fails back to initramfs.
I don't know if this is significant, but mount shows xvda1 mounted as /root, instead of /boot & I don't see the encrypted volumes:
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=944232k,nr_inodes=236058,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=195040k,mode=755)
/dev/xvda1 on /root type ext4 (ro,relatime,data=ordered)
(initramfs) lsblk
sh: lsblk: not found
(initramfs) ls -l /dev/xvd*
brw------- 1 202, 17 /dev/xvdb1
brw------- 1 202, 16 /dev/xvdb
brw------- 1 202, 5 /dev/xvda5
brw------- 1 202, 2 /dev/xvda2
brw------- 1 202, 1 /dev/xvda1
brw------- 1 202, 0 /dev/xvda
(domU) /etc/grub/grub.cfg:
(initramfs) cat grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
BEGIN /etc/grub.d/00_header
if [ -s $prefix/grubenv ]; then
set have_grubenv=true
load_env
fi
if [ "${next_entry}" ] ; then
set default="${next_entry}"
set next_entry=
save_env next_entry
set boot_once=true
else
set default="0"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function recordfail {
set recordfail=1
if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}
function load_video {
if [ x$feature_all_video_module = xy ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
if [ x$feature_default_font_path = xy ] ; then
font=unicode
else
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
font="/grub/unicode.pf2"
fi
if loadfont $font ; then
set gfxmode=auto
load_video
insmod gfxterm
set locale_dir=$prefix/locale
set lang=en_GB
insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
set timeout=30
else
if [ x$feature_timeout_style = xy ] ; then
set timeout_style=hidden
set timeout=10
Fallback hidden-timeout code in case the timeout_style feature is
unavailable.
elif sleep --interruptible 10 ; then
set timeout=0
fi
fi
END /etc/grub.d/00_header
BEGIN /etc/grub.d/05_debian_theme
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
if background_color 44,0,30,0; then
clear
fi
END /etc/grub.d/05_debian_theme
BEGIN /etc/grub.d/10_linux
function gfxmode {
set gfxpayload="${1}"
if [ "${1}" = "keep" ]; then
set vt_handoff=vt.handoff=1
else
set vt_handoff=
fi
}
if [ "${recordfail}" != 1 ]; then
if [ -e ${prefix}/gfxblacklist.txt ]; then
if hwmatch ${prefix}/gfxblacklist.txt 3; then
if [ ${match} = 0 ]; then
set linux_gfx_mode=keep
else
set linux_gfx_mode=text
fi
else
set linux_gfx_mode=text
fi
else
set linux_gfx_mode=keep
fi
else
set linux_gfx_mode=text
fi
export linux_gfx_mode
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
linux /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro quiet splash $vt_handoff
initrd /initrd.img-4.15.0-36-generic
}
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
menuentry 'Ubuntu, with Linux 4.15.0-36-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-36-generic-advanced-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
echo 'Loading Linux 4.15.0-36-generic ...'
linux /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro quiet splash $vt_handoff
echo 'Loading initial ramdisk ...'
initrd /initrd.img-4.15.0-36-generic
}
menuentry 'Ubuntu, with Linux 4.15.0-36-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-36-generic-recovery-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
recordfail
load_video
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
echo 'Loading Linux 4.15.0-36-generic ...'
linux /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro recovery nomodeset
echo 'Loading initial ramdisk ...'
initrd /initrd.img-4.15.0-36-generic
}
}
END /etc/grub.d/10_linux
BEGIN /etc/grub.d/20_linux_xen
END /etc/grub.d/20_linux_xen
BEGIN /etc/grub.d/20_memtest86+
menuentry 'Memory test (memtest86+)' {
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
knetbsd /memtest86+.elf
}
menuentry 'Memory test (memtest86+, serial console 115200)' {
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi
linux16 /memtest86+.bin console=ttyS0,115200n8
}
END /etc/grub.d/20_memtest86+
BEGIN /etc/grub.d/30_os-prober
END /etc/grub.d/30_os-prober
BEGIN /etc/grub.d/30_uefi-firmware
END /etc/grub.d/30_uefi-firmware
BEGIN /etc/grub.d/40_custom
This file provides an easy way to add custom menu entries. Simply type the
menu entries you want to add after this comment. Be careful not to change
the 'exec tail' line above.
END /etc/grub.d/40_custom
BEGIN /etc/grub.d/41_custom
if [ -f ${config_directory}/custom.cfg ]; then
source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
END /etc/grub.d/41_custom
(dom0) domu1.cfg:
Note: the initial kernel&ramdisk=/var/lib were for the network boot to perform the installation process as per instructions at Ubuntu Xen Guide
# cat domu1.cfg
# Guest name
name = "domu1"
128-bit UUID for the domain as a hexadecimal number.
Use "uuidgen" to generate one if required.
The default behavior is to generate a new UUID each time the guest is started.
#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
Kernel image to boot
#kernel = "/boot/vmlinuz"
#kernel = "/var/lib/xen/images/ubuntu-netboot/Ubuntu18.04-bionic-beaver/vmlinuz"
Ramdisk (optional)
#ramdisk = "/boot/initrd.gz"
#ramdisk = "/var/lib/xen/images/ubuntu-netboot/Ubuntu18.04-bionic-beaver/initrd.gz"
bootloader = "/usr/lib/xen-4.9/bin/pygrub"
Kernel command line options
extra = "root=/dev/xvda1"
Initial memory allocation (MB)
memory = 2048
Maximum memory (MB)
If this is greater than `memory' then the slack will start ballooned
(this assumes guest kernel support for ballooning)
maxmem = 6144
Number of VCPUS
vcpus = 7
Network devices
A list of 'vifspec' entries as described in docs/misc/xl-network-configuration.markdown
vif = [ 'bridge=xenbr0' ]
Disk Devices
A list of `diskspec' entries as described in docs/misc/xl-disk-configuration.txt
disk = [ '/dev/vg0/lv-1,raw,xvda,rw','/dev/vg0/lv-1swap,raw,xvdb,rw' ]
EDIT: (on dom0) the filesystems are readable & valid:
# guestfish -a /dev/vg0/lv-1
><fs> run
><fs> luks-open /dev/sda5 sda5_crypt
><fs> mount /dev/mapper/sda5_crypt /
><fs> mount /dev/sda1 /boot
><fs> ll /boot/grub
total 2388
drwxr-xr-x 5 0 0 4096 Oct 12 11:22 .
drwxr-xr-x 4 0 0 4096 Oct 12 11:23 ..
drwxr-xr-x 2 0 0 4096 Oct 12 11:22 fonts
-rw-r--r-- 1 0 0 712 Oct 12 11:08 gfxblacklist.txt
-r--r--r-- 1 0 0 6857 Oct 12 11:22 grub.cfg
-rw-r--r-- 1 0 0 1024 Oct 12 11:22 grubenv
drwxr-xr-x 2 0 0 12288 Oct 12 11:22 i386-pc
drwxr-xr-x 2 0 0 4096 Oct 12 11:22 locale
-rw-r--r-- 1 0 0 2397557 Oct 12 11:08 unicode.pf2
><fs> ll /initrd.img
-rw-r--r-- 1 0 0 56938107 Oct 12 11:23 /sysroot/boot/initrd.img-4.15.0-36-generic
EDIT 2:
To try and recover the crypt setup, I did the following:
losetup -f
losetup /dev/loop15 /dev/vg0/lv-1
kpartx -av /dev/loop15
udisksctl unlock -b /dev/mapper/loop15p5
mount /dev/dm-9 /mnt/d1
mount /dev/mapper/loop15p1 /mnt/d1/boot
ln -s /dev/mapper/luks-ba3b60bf-f06c-48d8-819a-1c2a2620252a /dev/mapper/xvda5
losetup -f
losetup /dev/loop16 /dev/vg0/lv-1swap
kpartx -av /dev/loop16
udisksctl unlock -b /dev/mapper/loop16p1
ln -s /dev/mapper/luks-e3a84c9d-f9e8-424d-a473-48af7dc22500 /dev/mapper/xvdb1
mount --bind /dev /mnt/d1/dev
mount --bind /sys /mnt/d1/sys
mount --bind /proc /mnt/d1/proc
chroot /mnt/d1
ln -s /dev/mapper/loop15p5 /dev/mapper/xvda5_crypt
ln -s /dev/mapper/loop16p1 /dev/mapper/xvdb1_crypt
cat /etc/crypttab
xvda5_crypt UUID=ba3b60bf-f06c-48d8-819a-1c2a2620252a none luks,discard
xvdb1_crypt UUID=e3a84c9d-f9e8-424d-a473-48af7dc22500 none luks,swap,discard
update-initramfs -k all -c
This last command resulted in the following output:
update-initramfs: Generating /boot/initrd.img-4.15.0-36-generic
Warning: /sbin/fsck.crypto_LUKS doesn't exist, can't install to initramfs, ignoring.
Continuing with update-grub, exiting chroot (without installing grub!), unmounting links & rebooting: I could not even reboot my host system, I had to boot from USB, fix grub and reboot. And still my domU cannot start (same failure back to initramfs), so presumably my actions amounted to nothing.
update-initramfsin my chrooted environment isWarning: /sbin/fsck.crypto_LUKS doesn't exist, can't install to initramfs, ignoring.- see EDIT 2 for details in my main post. – asoundmove Oct 15 '18 at 14:42/sbin/fsck.crypto_LUKScan't be added. Did you also runupdate-grubafter runningupdate-initramfs -k all -cwhile in the chroot? If so, the issues may reside with the Xen portion of all this which, unfortunately, I can't help with. – b_laoshi Oct 16 '18 at 00:27