0

I have an Ubuntu machine that has been through several kernel upgrades. At the start of the day, I had 3 kernels installed: 5.11.0-34, 5.11.0-46, and 5.11.0-49. I had to upgrade a bunch of packages, and afterward took the opportunity to remove the middle kernel to open up room in my boot partition.

Now, I cannot get either remaining kernel to boot. Neither of them prompts for the password to decrypt the drive where Linux is installed. It doesn't matter whether I boot into recovery mode or not, they print messages and eventually drop into a shell like this:

Unable to init MCE device (rc: -5)
Volume group "vgubuntu" not found
Cannot process volume group vgubuntu
Gave up waiting for suspend/resume device
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/vgubuntu-root does not exist.  Dropping to a shell!

BusyBox v1.30.1 (Ubuntu 1:1.30.1-6ubuntu2.1) built-in shell (ash)

I have found a similar account here, but I don't have any devices in /dev with a name prefixed with sda. I'm guessing that's because I don't have any SATA drives. My drives are all NVMe, and I don't know which one has the root file system.

Long ago I added mce=off as a kernel parameter. It is present in every GRUB menu option.

How can I fix my installation to boot?

John Freeman
  • 101
  • 1
    Ubuntu 21.04 (along with all flavors) is End-of-Life and thus unsupported on this site (https://askubuntu.com/help/on-topic), and many other Ubuntu sites, unless your question is specific to moving to a supported release of Ubuntu. https://fridge.ubuntu.com/2022/01/21/ubuntu-21-04-hirsute-hippo-end-of-life-reached-on-january-20-2022/ https://help.ubuntu.com/community/EOLUpgrades – guiverc Jan 28 '22 at 20:59
  • Ok? How can I boot into anything? I don't care which version. – John Freeman Jan 28 '22 at 21:12
  • Did you just stop reading at 21.04 and spam your copypasta? How can I upgrade if I can't even boot into it? – John Freeman Jan 28 '22 at 21:15
  • 1
    I would stop trying to boot your system, and explore what is there via live media; even though disk is encrypted (using unknown method as you didn't stipulate & through many releases reminds me of my box with no-longer-offered-by-default-encryption that requires addition of a package) so you can backup any data you don't have good backups of, then re-install a supported release over the existing encrypted install (that works; I did the re-install instead of release-upgrade for QA-test purposes.. my install only needed package as my encryption isn't the default method as already stated) – guiverc Jan 28 '22 at 21:17
  • It's not many releases. How many encryption methods has Ubuntu been through? This machine started at version 19.10, so it's using whatever method was available during that installation. – John Freeman Jan 28 '22 at 21:23
  • 1
    Your question as written is purely about an off-topic 21.04 problem (busybox version isn't supported etc).. No mention is made of anything on-topic (ie. no mention there of your need/attempt to release-upgrade to a supported release.. so you're off-topic here (if you take my hint; if you want help here make your question on-topic as per first comment!) – guiverc Jan 28 '22 at 21:28
  • What a troll. Any excuse to be unhelpful. Trying to make me jump through rhetorical hoops to satisfy your power trip. If you can't help, then just move on. It's very clear what my problem is, and that it's not version specific. If you read through my question history, boot problems are recurring problem with every version of Linux. Regardless, I'm open to all solutions. I'm not asking how to install 21.04. I just want to understand what makes the boot sequence skip over decrypting the drive, and how to fix it. – John Freeman Jan 28 '22 at 21:32
  • "Busybox version isn't supported" as if I asked about how to use Busybox? – John Freeman Jan 28 '22 at 21:36
  • 1
    Your paste shows an unsupported OS or release is being used; as the busybox in your paste is not from a supported release of Ubuntu. You can re-install a later version of Ubuntu; ie. I did re-installs of later releases (20.04) over older releases (18.04) just to complete QA-tests; despite my boxes having no-longer-default encryption; results were good - no user file touched, my additional apps were re-installed automatically (they were available as I checked in new release) .. ie. my comment was geared to re-install a supported OS (20.04 or 21.10!). https://askubuntu.com/help/on-topic – guiverc Jan 28 '22 at 21:42
  • You are such a troll, dude. You can't just be helpful. Instead, you have to play policeman and show off that you know how to fix the problem but won't teach me how. Go harass someone else. – John Freeman Jan 28 '22 at 21:46
  • 1
    The purpose of my repeated comments (after first one which showed my off-topic voting reason) was to get you to read the rules for the site & my hints on how to make your question on-topic; allowing me to retract my close-vote & remove the comments... It was not intended to troll; but bring you within site rules.. Even not mentioning 21.04 as per your edit; you still mention off-topic kernel versions (5.11 kernel is EOL in Ubuntu, no on-topic release uses it), off-topic package versions (busybox) etc. I also gave clues as to how I'd explore fixing it.. (but minimal as you're off-topic) – guiverc Jan 29 '22 at 07:20
  • "give you hints" "gave clues" Do you understand how unhelpful and antagonizing it is to basically say "I can help you, but I'm not going to unless you phrase your question in a special way that I'm not going to tell you exactly. Here's a hint. Wink wink nudge nudge"? Get off your high horse. This is supposed to be a site for helping people. If you're unhappy with the form of my question, then edit it. Did you miss my comment where I asked "how can I upgrade"? So even after I asked your special question, you continued to troll me and be unhelpful. – John Freeman Jan 30 '22 at 13:57
  • Be direct. Say what you really mean. Stop with the word games, hints, clues, whatever you want to call it. "Please edit this question to say ..., and I'll work on posting an answer to that in the mean time." How hard is that? – John Freeman Jan 31 '22 at 00:04
  • One last thing: You come across like "Oooo, that version, less than a year old, is one week end of life, Gotcha! Haha! I'm not going to help you now! Too late!" That attitude stinks. It's very unwelcoming. – John Freeman Jan 31 '22 at 00:09
  • 1
    @John We can't help with problems on an unsupported release. Standard releases come out every 6 months with 9 months of support. If you want to use the standard release cadence, you are expected to upgrade before your release goes EOL. If you wait too long, it can become difficult/impossible to fix problems since there are no updates. It is also our policy to not answer questions about releases that have passed the EOL date. Your conduct here is not becoming of our code of conduct. Install a supported release. If you still have the problem, you can ask. – Nmath Feb 01 '22 at 22:45
  • EOL releases are also harder to support; as to look up details we can use online which show only supported releases; ie. hirsute isn't there, or CLI tools which show again only supported releases PLUS ESM releases but again hirsute isn't available. Without this wealth of package, depends etc details, support is much more difficult, requiring access to a hirsute machine to check facts/theories etc. You're making a much more significant request due effort required to check details when asking about unsupported releases and expecting others to provide support for EOL products. – guiverc Feb 01 '22 at 23:04
  • I didn't ask for support specific to any version. I asked how to get access back to my encrypted partition. I was open to installing a new version. I didn't care what version the solution came in. See the answer I posted. I ran a live USB with the latest version of Ubuntu to implement that solution. Anyone here could have offered the same answer to help, but no one did. Instead, I got turned away and rapped on the hand for mentioning the version who cannot be named. I had a problem. No one here attempted to address the problem but focused on the version number that I provided only for context. – John Freeman Feb 03 '22 at 00:07

1 Answers1

0

Something got borked somewhere and I had to run update-initramfs. I found very similar instructions in three separate places:

  1. https://ubuntuforums.org/showthread.php?t=2409754&s=e1f324bf5e566b3bb93374cd07bdcc17&p=13828993
  2. https://askubuntu.com/a/868726/538768
  3. https://feeding.cloud.geek.nz/posts/recovering-from-unbootable-ubuntu-encrypted-lvm-root-partition/

Here's how I got there.

I loaded Ubuntu off a live USB and ran fdisk -l to see my partitions and guess which one was encrypted. I saw these (among others):

  • /dev/nvme2n1p1: 512M EFI System
  • /dev/nvme2n1p2: 732M Linux filesystem
  • /dev/nvme2n1p3: 1.8T Linux filesystem <-- I guessed it was this one.

Then I decrypted the partition and mounted it like this:

sudo -i
cryptsetup open /dev/nvme2n1p3 $name
vgchange -ay
mkdir /mnt/root
mount /dev/mapper/$name /mnt/root

That let me inspect /etc/crypttab to see which device name to use when decrypting the partition (nvme0n1p3_crypt in this case):

nvme0n1p3_crypt UUID=743ab129-75bb-429b-8366-9c066f00c4fe none luks,discard

Then I looked at /etc/fstab to see which partitions were the boot partition and EFI partition:

# /boot was on /dev/nvme0n1p2 during installation
UUID=773ceeb2-5c0f-4838-baad-a1182d7fdd80 /boot           ext4    defaults        0       2
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=5C17-FB32  /boot/efi       vfat    umask=0077      0       1

At installation, these partitions were named like nvme0n1p*, but no longer. I could find their current names by listing /dev/disk/by-uuid:

$ ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root 15 Jan 31 12:29 5C17-FB32 -> ../../nvme2n1p1
lrwxrwxrwx 1 root root 15 Jan 31 12:29 743ab129-75bb-429b-8366-9c066f00c4fe -> ../../nvme2n1p3
lrwxrwxrwx 1 root root 15 Jan 31 12:29 773ceeb2-5c0f-4838-baad-a1182d7fdd80 -> ../../nvme2n1p2

Now I had all the pieces I needed to follow the instructions. Here are the actual commands I executed:

sudo -i
cryptsetup open /dev/nvme2n1p3 nvme0n1p3_crypt
mount /dev/mapper/nvme0n1p3_crypt /mnt/root
mount /dev/nvme2n1p2 /mnt/root/boot
mount /dev/nvme2n1p1 /mnt/root/boot/efi
mount --bind /dev /mnt/root/dev
mount --bind /run /mnt/root/run
chroot /mnt/root
mount -t proc proc /proc
mount -t sysfs sys /sys
update-initramfs -c -k all

Then I was able to restart the machine and boot into one of the installed kernels.

John Freeman
  • 101