lvmcache failed to boot: "device-mapper: cache-policy: unknown policy type"

Question

TL;TR

Failed to boot with lvm2's new default cache-pool setting cache-policy-smq

The problem

I have created a lvm disk with lvmcache on SSD drive on ubuntu 16.04 Following this, but failed to mount my root volume after reboot server.

I can boot into 16.04 live CD and mount /dev/mapper/vg0-root successfully (with Boot-repair, mdadm, thin-provisioning-tools).

After some digging, It seems lvmcache's new policy dm-cache-smq is missing/unsupported in somewhere, but i have no idea what to do next.

How can i fix this?

Error messages on boot

The error message repeat showing until mount wait timeout, then console drops a (initramfs) shell.

Begin: Running /scripts/local-block . . . lumetad is not active yet, using direct activation during sysinit
/sbin/nodprobe failed: 1
  device-mapper: cache: You have created a cache device with a lot of individual cache blocks (7600000)
  All these mappings can consume a lot oF kernel memory, and take some time to read/write
  Please consider increasing the cache block size to reduce the overall cache block count
  device-mapper: cache-policy: unknown policy type
  device-mapper: table: 252:4: cache: Error creating cache's policy
  device-napper: ioctl: error adding target to table
  device-mapper: reload ioctl on (252:4) failed: Invalid argument
done.

Settings i have now

/etc/initramfs-tools/modules

dm_cache
dm_cache_mq
dm_cache_smq
dm_persistent_data
dm_bufio

/etc/initramfs-tools/hooks/lvmcache

#!/bin/sh
PREREQ="lvm2"
prereqs()
{
  echo "$PREREQ"
}
case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac
if [ ! -x /usr/sbin/cache_check ]; then
  exit 0
fi
. /usr/share/initramfs-tools/hook-functions
copy_exec /usr/sbin/cache_check
manual_add_modules dm_cache dm_cache_mq dm_cache_smq dm_persistent_data dm_bufio

Workaround

I change lvmcache back to mq policy to get my server online temporarily, but still looking for solutions for cache with smq policy.

lvremove vg0/cachedata
lvcreate -L 480M -n cachemeta vg0 /dev/sde
lvcreate -L 475000M -n cachedata vg0 /dev/sde
# Use mq here
lvconvert --type cache-pool --cachepolicy mq --chunksize 8192 --poolmetadata vg0/cachemeta --cachemode writeback vg0/cachedata --yes
lvconvert --type cache --cachepool vg0/cachedata vg0/root

References

How to install Ubuntu with both disk encryption AND SSD caching
Boot-Repair
Bug 1247386 - lvcreate and lvchange need '--cachepolicy' validity checks

If you solved your problem yourself, please answer your own question and accept it. Don’t put the answer in your question or the comments! Send me a message in the comments if you want my up-vote. :-) — David Foerster, Feb 01 '17 at 21:56

score 3 · Accepted Answer · answered Feb 06 '17 at 11:18

This is probably late as an answer, but I did the things in your post (but with a ubuntu 16.04.1 installation), and it worked with the smq-policy. It seems the issue you had has been resolved.

This is what I did, command by command.

First, I had a big harddisk on /dev/sda and an SSD on /dev/sdb.

I formatted the /dev/sda to have 10MB free in the beginning, then followed by a 1GB ext4 primary partition, which will be used for /boot in the installation (you cannot cache the kernel, so kernels can go in there). The rest of /dev/sda is cleared as sda2.

sudo pvcreate /dev/sda2
sudo pvcreate /dev/sdb
sudo vgcreate VG /dev/sda2 /dev/sdb
sudo lvcreate -L 3.5T -n lv VG /dev/sda2
sudo lvcreate -L 450G -n lv_cache VG /dev/sdb
sudo lvcreate -L 4.5G -n lv_cache_meta VG /dev/sdb
sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata VG/lv_cache_meta VG/lv_cache
sudo lvs -a
sudo lvconvert --type cache --cachepool VG/lv_cache VG/lv
sudo lvs -a
sudo lvdisplay
sudo mkfs.ext4 /dev/VG/lv

On this point, I install ubuntu on the /dev/VG/lv partion as /, with /boot on /dev/sda1.

After installation, the kernel of this new installation needs to be recompiled with some modifications (in case of ubuntu 16.04.1)

#### mount existing installation
sudo chmod 777 /var/cache/app-info/xapian/default -R
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install thin-provisioning-tools
sudo vgchange -a y VG
sudo mkdir /new
sudo nano /etc/fstab
###add this line to fstab: /dev/VG/lv /new ext4 defaults 0 0
sudo mount /dev/VG/lv

### We will need to install some new packages in the chroot, so fix internet
sudo mount --bind /dev /new/dev
sudo mount --bind /proc /new/proc
sudo mount --bind /sys /new/sys
sudo cp /etc/resolv.conf /new/etc/resolv.conf
sudo chroot /new
echo 'nameserver 8.8.4.4' | sudo tee -a /etc/resolv.conf
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install thin-provisioning-tools

#### recompile kernel with additional hook for dm_cache_smq
echo "dm_cache" >> /etc/initramfs-tools/modules
echo "dm_cache_mq"  >> /etc/initramfs-tools/modules
echo "dm_cache_smq"  >> /etc/initramfs-tools/modules
echo "dm_persistent_data"  >> /etc/initramfs-tools/modules
echo "dm_bufio"  >> /etc/initramfs-tools/modules

nano /etc/initramfs-tools/hooks/cache_hook

contents from here: http://forums.debian.net/viewtopic.php?f=5&t=119644 The last line has been modified for the smq to work

#!/bin/sh

PREREQ="lvm2"

prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

if [ ! -x /usr/sbin/cache_check ]; then
    exit 0
fi

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check

manual_add_modules dm_cache dm_cache_mq dm_cache_smq dm_persistent_data dm_bufio

Then we continue with the following to compile the kernel and put it in the correct spot.

chmod +x /etc/initramfs-tools/hooks/cache_hook
mkdir /realboot
mount /dev/sda1 /realboot
update-initramfs -v -u -k all -b /realboot

Now reboot, and it worked. I tried this on 2 machines already, more to follow.

score 0 · Answer 2 · answered Mar 08 '20 at 21:29

To make the situation a bit more interesting, my root file system is on an lvm volume on an md raid 5 array. Writeback caching the root volume on an ssd disk rendered the system unbootable, stopping in busybox, where I need to read the above solution carefully to fix this. To restore booting, I started the system with a Ubuntu startup USB stick, installed mdadm to start the md array, added universe to /etc/apt/sources.list to be able to install thin-provisioning-tools and removed caching on the root volume:

Boot from Ubuntu USB startup stick
# sudo edit /etc/apt/sources.list # add "universe" to all deb lines
# sudo apt update
# sudo apt install -y mdadm
# sudo mdadm --assemble /dev/md0
# sudo apt install -y thin-provisioning-tools
# sudo lvm
lvm> lvconvert --splitcache vg/root
Logical volume vg/root is not cached and cache pool vg/CacheRoot is unused.
# exit
# reboot

Successful boot to uncached root and ready for next trial.