ZFS rpool fails to mount on boot

Question

My system was booting fine, but I was trying to resolve a couple ZFS related boot errors that said:

cannot mount '/root': directory is not empty
cannot mount '/var/cache': directory is not empty

In my infinite wisdom, I added "rpool/var/cache /var/cache zfs defaults 0 0" to my fstab, and my system failed to reboot and it kicked me to terminal where I was able to remove that line from fstab and reboot my system. Here's the current state of /etc/fstab:

rpool/var/log /var/log zfs defaults 0 0
rpool/var/tmp /var/tmp zfs defaults 0 0
/dev/zvol/rpool/swap none swap defaults 0 0

One weird thing, after it rebooted, it was stuck in a login loop and I had to go to terminal and chown my username folder in the home directory per these instructions.

Now rpool will not mount and, as a result, my folders are missing from my home directory, along with all my files, since a lot of my system folders are inherited from rpool. Here is what my ZFS looks like:

$ zfs get mountpoint,mounted,canmount,overlay rpool -t filesystem -r
NAME               PROPERTY    VALUE                SOURCE
rpool              mountpoint  /                    local
rpool              mounted     no                   -
rpool              canmount    on                   local
rpool              overlay     off                  default
rpool/ROOT         mountpoint  none                 local
rpool/ROOT         mounted     no                   -
rpool/ROOT         canmount    off                  local
rpool/ROOT         overlay     off                  local
rpool/ROOT/ubuntu  mountpoint  /                    local
rpool/ROOT/ubuntu  mounted     yes                  -
rpool/ROOT/ubuntu  canmount    noauto               local
rpool/ROOT/ubuntu  overlay     off                  inherited from rpool/ROOT
rpool/home         mountpoint  /home                inherited from rpool
rpool/home         mounted     no                   -
rpool/home         canmount    on                   default
rpool/home         overlay     off                  default
rpool/home/greg    mountpoint  /home/greg           inherited from rpool
rpool/home/greg    mounted     yes                  -
rpool/home/greg    canmount    on                   default
rpool/home/greg    overlay     off                  default
rpool/home/root    mountpoint  /root                local
rpool/home/root    mounted     no                   -
rpool/home/root    canmount    on                   default
rpool/home/root    overlay     off                  default
rpool/srv          mountpoint  /srv                 inherited from rpool
rpool/srv          mounted     yes                  -
rpool/srv          canmount    on                   default
rpool/srv          overlay     off                  default
rpool/var          mountpoint  /var                 inherited from rpool
rpool/var          mounted     no                   -
rpool/var          canmount    off                  local
rpool/var          overlay     off                  default
rpool/var/cache    mountpoint  /var/cache           inherited from rpool
rpool/var/cache    mounted     yes                  -
rpool/var/cache    canmount    on                   default
rpool/var/cache    overlay     off                  default
rpool/var/games    mountpoint  /var/games           inherited from rpool
rpool/var/games    mounted     yes                  -
rpool/var/games    canmount    on                   default
rpool/var/games    overlay     off                  default
rpool/var/log      mountpoint  legacy               local
rpool/var/log      mounted     yes                  -
rpool/var/log      canmount    on                   default
rpool/var/log      overlay     off                  default
rpool/var/mail     mountpoint  /var/mail            inherited from rpool
rpool/var/mail     mounted     yes                  -
rpool/var/mail     canmount    on                   default
rpool/var/mail     overlay     off                  default
rpool/var/spool    mountpoint  /var/spool           inherited from rpool
rpool/var/spool    mounted     yes                  -
rpool/var/spool    canmount    on                   default
rpool/var/spool    overlay     off                  default
rpool/var/tmp      mountpoint  legacy               local
rpool/var/tmp      mounted     yes                  -
rpool/var/tmp      canmount    on                   default
rpool/var/tmp      overlay     off                  default

All my pools are in my zpool list:

$ zpool list
NAME         SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Pool-1tb     976G   414G   562G         -     4%    42%  1.00x  ONLINE  -
Pool-500gb   488G  70.1G   418G         -     0%    14%  1.00x  ONLINE  -
rpool        952G   684G   268G         -    57%    71%  1.00x  ONLINE  -

I'm unable to mount rpool after booting:

$ zfs mount rpool
cannot mount '/': directory is not empty

Here's the current ZFS related error when I run journalctl -b:

Any suggestions on how to get rpool to mount on boot and how to get my system back?

Running Ubuntu 18.04 with zfsutils-linux 0.7.5 and zfs-dkms 0.7.5

EDIT:

The output of df -h on /

$ ls -la
total 174
drwxr-xr-x   26 root root   30 Apr 29 06:58 .
drwxr-xr-x   26 root root   30 Apr 29 06:58 ..
drwxr-xr-x    2 root root  166 Apr 29 06:56 bin
drwxr-xr-x    4 root root   22 May  5 06:21 boot
drwxr-xr-x   24 root root 5380 May  5 07:22 dev
drwxr-xr-x  154 root root  265 May  5 07:24 etc
drwxr-xr-x    3 root root    3 May  4 12:21 home
lrwxrwxrwx    1 root root   33 Apr 29 06:58 initrd.img -> boot/initrd.img-4.15.0-99-generic
lrwxrwxrwx    1 root root   33 Apr 29 06:58 initrd.img.old -> boot/initrd.img-4.15.0-96-generic
drwxr-xr-x   24 root root   37 Apr 20 18:58 lib
drwxr-xr-x    2 root root   45 Jan 20  2019 lib32
drwxr-xr-x    2 root root    3 Jul 10  2019 lib64
drwxr-xr-x    3 root root    3 May  3 01:32 media
drwxr-xr-x    2 root root    2 Apr 27  2018 mnt
drwxr-xr-x   13 root root   13 Mar 29 12:34 opt
drwxr-xr-x    5 greg root    5 Apr 23 11:42 Pool-1tb
drwxr-xr-x    6 greg root    6 Apr 21 14:17 Pool-500gb
dr-xr-xr-x 1325 root root    0 May  5 07:21 proc
drwxr-xr-x   11 root root   12 Apr 20 19:27 root
drwxr-xr-x   37 root root 1140 May  5 07:22 run
drwxr-xr-x    2 root root  218 May  2 17:25 sbin
drwxr-xr-x   16 root root   17 Apr 30 13:02 snap
drwxr-xr-x    2 root root    2 Apr 27  2018 srv
dr-xr-xr-x   13 root root    0 May  5 07:21 sys
drwxrwxrwt   20 root root   21 May  5 07:31 tmp
drwx------    4 root root    4 Apr 28  2018 .Trash-0
drwxr-xr-x   12 root root   12 Jul 27  2019 usr
drwxr-xr-x   15 root root   17 Apr 27  2018 var
lrwxrwxrwx    1 root root   30 Apr 29 06:58 vmlinuz -> boot/vmlinuz-4.15.0-99-generic
lrwxrwxrwx    1 root root   30 Apr 29 06:58 vmlinuz.old -> boot/vmlinuz-4.15.0-96-generic

The output of ls -la on /home

$ ls -la
total 18
drwxr-xr-x  3 root root  3 May  4 12:21 .
drwxr-xr-x 26 root root 30 Apr 29 06:58 ..
drwxr-xr-x 22 greg greg 25 May  5 08:46 greg

The output of ls -la on /home/greg

$ df -h
Filesystem         Size  Used Avail Use% Mounted on
udev                95G     0   95G   0% /dev
tmpfs               19G  4.6M   19G   1% /run
rpool/ROOT/ubuntu  258G   24G  235G  10% /
tmpfs               95G  4.0K   95G   1% /dev/shm
tmpfs              5.0M  4.0K  5.0M   1% /run/lock
tmpfs               95G     0   95G   0% /sys/fs/cgroup
rpool/var/log      236G  1.4G  235G   1% /var/log
rpool/var/tmp      235G  128K  235G   1% /var/tmp
/dev/loop0         161M  161M     0 100% /snap/gnome-3-28-1804/116
/dev/loop3          94M   94M     0 100% /snap/core/9066
/dev/loop1          94M   94M     0 100% /snap/darktable-kyrofa/49
/dev/loop4         243M  243M     0 100% /snap/gnome-3-34-1804/27
/dev/loop2         141M  141M     0 100% /snap/gnome-3-26-1604/97
/dev/loop5         116M  116M     0 100% /snap/checksum-validator/2
/dev/loop6         141M  141M     0 100% /snap/gnome-3-26-1604/98
/dev/loop7         157M  157M     0 100% /snap/gnome-3-28-1804/110
/dev/loop8         2.5M  2.5M     0 100% /snap/gnome-calculator/730
/dev/loop9          93M   93M     0 100% /snap/darktable-kyrofa/45
/dev/loop10         44M   44M     0 100% /snap/snap-store/415
/dev/loop11         94M   94M     0 100% /snap/core/8935
/dev/loop12         63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop13        143M  143M     0 100% /snap/slack/23
/dev/loop14        143M  143M     0 100% /snap/slack/22
/dev/loop15         55M   55M     0 100% /snap/core18/1705
/dev/loop16         62M   62M     0 100% /snap/authy/3
/dev/loop17         55M   55M     0 100% /snap/gtk-common-themes/1502
/dev/loop18        9.2M  9.2M     0 100% /snap/canonical-livepatch/95
/dev/loop19         55M   55M     0 100% /snap/core18/1754
/dev/loop20        4.4M  4.4M     0 100% /snap/gnome-calculator/704
Pool-1tb           946G  415G  532G  44% /Pool-1tb
Pool-500gb         473G   71G  403G  15% /Pool-500gb
rpool/srv          235G  128K  235G   1% /srv
rpool/var/games    235G  128K  235G   1% /var/games
rpool/var/mail     235G  128K  235G   1% /var/mail
tmpfs               19G   16K   19G   1% /run/user/120
tmpfs               19G   36K   19G   1% /run/user/1000

The output of ls -la on /

$ ls -la
ls: cannot access '.gvfs': Permission denied
total 69
drwxr-xr-x 23 greg greg   26 May  5 07:22 .
drwxr-xr-x  3 root root    3 May  4 12:21 ..
-rw-------  1 greg greg 2938 May  5 07:24 .bash_history
drwx------ 18 greg greg   20 May  5 05:57 .cache
drwx------ 17 greg greg   20 May  4 17:49 .config
drwx------  3 root root    3 May  4 13:07 .dbus
drwxr-xr-x  2 greg greg    2 May  4 13:04 Desktop
drwxr-xr-x  2 greg greg    2 May  4 13:04 Documents
drwxr-xr-x  2 greg greg    2 May  4 13:04 Downloads
drwx------  2 greg greg    2 May  4 13:04 .gconf
drwx------  3 greg greg    5 May  4 13:06 .gnupg
d?????????  ? ?    ?       ?            ? .gvfs
-rw-------  1 greg greg 2480 May  5 07:22 .ICEauthority
drwx------  3 greg greg    3 May  4 13:04 .local
drwx------  5 greg greg    5 May  4 17:49 .mozilla
drwxr-xr-x  2 greg greg    2 May  4 13:04 Music
drwxr-xr-x  2 greg greg    2 May  4 12:21 NAS
drwx------  3 greg greg    3 May  4 13:04 .nv
drwxr-xr-x  2 greg greg    3 May  5 05:57 Pictures
drwx------  3 greg greg    3 May  4 17:49 .pki
drwxr-xr-x  2 greg greg    2 May  4 13:04 Public
drwxr-xr-x  3 greg greg    3 May  4 13:06 snap
drwx------  2 greg greg    2 May  4 13:06 .ssh
-rw-r--r--  1 greg greg    0 May  4 13:07 .sudo_as_admin_successful
drwxr-xr-x  2 greg greg    2 May  4 13:04 Templates
drwxr-xr-x  2 greg greg    2 May  4 13:04 Videos

The output of ls -la on /root

$ ls -la
total 42
drwxr-xr-x 11 root root  12 Apr 20 19:27 .
drwxr-xr-x 26 root root  30 Apr 29 06:58 ..
-rw-------  1 root root 399 May  4 12:59 .bash_history
drwx------  4 root root   4 Apr 28  2018 .cache
drwx------  6 root root   6 Jun 23  2018 .config
drwx------  3 root root   3 Apr 28  2018 .dbus
drwx------  2 root root   3 Apr 20 20:01 .ecryptfs
drwx------  2 root root   2 Nov  8  2018 .gconf
drwxr-xr-x  2 root root   3 May  5 07:21 .hplip
drwxr-xr-x  3 root root   3 Apr 28  2018 .local
drwxr-xr-x  3 root root   3 Mar 29 19:01 snap
drwx------  3 root root   6 May  5 06:24 .synaptic

The output of ls -la on /var/cache

$ ls -la
total 142
drwxr-xr-x 18 root root  19 Feb 18 06:30 .
drwxr-xr-x 15 root root  17 Apr 27  2018 ..
drwx------  2 root root  22 Apr 30 13:02 apparmor
drwxr-xr-x  4 root root   4 Dec  6  2018 app-info
drwxr-xr-x  3 root root   5 May  5 06:21 apt
drwxr-xr-x  2 root root   6 Apr 29  2018 cracklib
drwxrwx---  3 root lp    11 May  5 06:31 cups
drwxr-xr-x  2 root root   7 May  5 06:21 debconf
drwxr-xr-x  2 root root 153 Apr 30 13:02 fontconfig
drwxr-xr-x  2 root root   4 May  4 13:05 fwupd
drwxr-xr-x  2 root root   2 Jun  5  2018 gdm
drwx------  2 root root   3 May  2 17:26 ldconfig
drwxr-xr-x 35 man  man   37 May  5 06:21 man
-rw-r--r--  1 root root 227 May  5 05:28 motd-news
drwxr-xr-x  3 root root   3 Sep  3  2018 PackageKit
drwxr-xr-x  3 root root   5 May  5 07:22 samba
drwx------  2 root root   2 Aug 30  2016 slack
drwxr-xr-x  3 root root   6 May  4 15:28 snapd
drwxr-xr-x  2 root root   3 Apr 29  2018 ubuntu-advantage-tools

The output of ls -la on /var/spool

$ ls -la
total 11
drwxr-xr-x  5 root   root  5 May  4 12:26 .
drwxr-xr-x 15 root   root 17 Apr 27  2018 ..
drwxr-xr-x  3 root   root  3 May  4 12:26 cron
drwx--x---  3 root   lp    3 May  4 12:26 cups
drwx------  2 syslog adm   2 May  4 12:21 rsyslog

Apparently you are trying to mount zfs on mountpoints (directories) that are not empty. After booting with the error, what is the output of df -h and what are the content of /, /home, /home/greg, /root, /var/cache and /var/spool ? Do an ls -la on each of them, and update your question with the results - basically we just need to see if each directory contains files or not (except from . and ..). — Soren A, May 05 '20 at 11:18
@SorenA I've updated my post with the information you requested. — guttermonk, May 05 '20 at 13:59
It looks like the right zfs volumes are mounted. What are the content of /etc/fstab ? — Soren A, May 05 '20 at 14:18
@SorenA I have the contents of fstab listed above. (second code snippet from the top) — guttermonk, May 05 '20 at 14:21
@SorenA if you look at the output for "zfs get mountpoint,canmount,mounted rpool -t filesystem -r" it says that rpool is not mounted. (third snippet down from the top) — guttermonk, May 05 '20 at 14:29
rpool and rpool/ROOT/ubuntu has same mountpoint / , so only one can be mounted. If it was my system,I would boot it in rescue mode (single user mode) unmount everything execpt from /. Then work through each mount, verifying every mountpoint directory IS empty. If not,fix it, and mount that one ... and so on. And start by making a good backup, if you end up deleting something needed. and verify that you can restore from it, before you need it. — Soren A, May 05 '20 at 14:42
@SorenA Thanks for the idea! I was able to delete the home folder in my active session, and then link my ZFS home folder with a legacy mount, and turned canmount off for everything else (since a lot of those folders were causing errors on boot anyway). Everything appears to be working for now, but if I have any problems I'll probably end up doing what you suggested. I appreciate you taking the time to help. — guttermonk, May 06 '20 at 15:12

Amitie 10g · Answer 1 · 2020-06-13T22:27:48.140

You need to remove the redundant, unused mountpoint attribute from the datasets. If rpool (the root dataset) is actually unused, remove the mountpoint attribute and set it on the dataset actually used (rpool/ROOT/ubuntu):

zfs set mountpoint=none rpool

And the most important, you should set the rootfs zpool attribute in addition to set the mountpoint=/ at the dataset level:

zpool set bootfs=rpool/ROOT/ubuntu rpool

Also, is safe to delete the mountpoint directories (ZFS will create them when mounting). Please rename or backup those directories.

If you don't have a rescue disk, you may get the Busybox shell by appending init=/bin/Busybox sh and remove the root= argument in the boot command line arguments. Then, you may import the pool without mounting (zpool import -R /mnt -n rpool), then, edit the attributes as you need.

Finally, never rely on fstab for mounting zfs filesystem datasets (except for zvols like swap), unless you're know what you doing, as datasets with standard mountpoint attribute are mounted early. Use mountpoint=legacy to allow be mounted using mount or trough fstab.

ZFS rpool fails to mount on boot

1 Answers1