The issue turns out to have multiple answers. Here's what you should do:
- Check
journalctl -xr |grep -iA50 nvidia
. A50 means you are looking for 50 lines before the fail event. It's the best way to triangulate the error.
In my case, nvidia caused 4 issues. Each emerges after I fix the one above it.
- Persistence daemon service file in
/lib/systemd/system/nvidia-persistenced.service
has a line called Before=systemd-backlight@backlight:nvidia_0.service
. For some reasons systemd can't parse this line.
- I have swap on a different disk. fstab and fsck doesn't wait long enough to mount the
/
and swap partition so it hangs or go into emergency mode. I changed the timeout for mount x-systemd.mount-timeout=5s
and it passes.
- Apparently verbose logging of systemd also causes boot to go into emergency mode. Probably an ubuntu bug.
- Multiple services fail after startup finished (Cisco VPN, libvirtd, NetworkManager).
In all cases, I can't get a TTY with Ctrl + Alt + F2
. The hint is the services fail and hang after this line:
systemd[1]: Startup finished in 4.324s (kernel) + 21.391s (userspace) = 25.716s.
systemd-analyze blame
shows nothing is causing the abnormal wait time. I interpreted it as the stem finished starting up but somehow couldn't get to the login screen, and that means gdm or lightdm has issue. I have gdm.
I just google gdm nvidia
, read /var/log/Xorg.0.log
and found the some links to Xorg and Wayland issue. I ended up following this link: https://linuxconfig.org/how-to-disable-wayland-and-enable-xorg-display-server-on-ubuntu-18-04-bionic-beaver-linux
This solution doesn't have any nvidia keyword, so google can't find it: WaylandEnable=false
If these don't work. Just press Ctrl+Alt+Del to repeatedly trigger a reboot. It'll work eventually. Less work to fix it permanently and doesn't physically damage your HDD.
Edit: the gdm issue returned. I changed the run-level to 5 (run-level 3 + display manager). I haven't seen an issue since except it takes a bit longer to get gdm up. This also fixed the DVXK issue after Ubuntu returns from sleep.
Lesson: google only helps you solve specific issues. If your problem touches multiple issues then you need to learn to read the system log and fix one by one. See how many tag I put on this question :D could have added more: xorg, gdm, fstab, swap, fsck