0

Update: With some hints from Terrence and heynnema, this problem is resolved. The update to kernel 5.4.0-52 was available, so I removed the kernel 51 files from /boot (carefully leaving the kernel 48 files in place) and allowed the updater to run. The system booted on the 52 kernel just fine.

I'm running an AMD Ryzen 2400G on an Asus PRIME X-470 Pro motherboard. A recent update gave me 5.4.0-51-generic, and the system no longer boots, coming up instead to BusyBox and the initramfs> prompt.

Falling back to 5.4.0-48-generic works.

Now there's another software update available, but I need to hang onto all of the 5.4.0-48-generic stuff in /boot in case 5.4.0-whatevercomesnext-generic doesn't boot, either.

# dmidecode -s bios-version
5603
root@deepsky:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           60Gi       2.0Gi        56Gi        81Mi       2.6Gi        58Gi
Swap:          62Gi          0B        62Gi
root@deepsky:~# lshw -c memory
  *-firmware                
       description: BIOS
       vendor: American Megatrends Inc.
       physical id: 0
       version: 5603
       date: 07/28/2020
       size: 64KiB
       capacity: 16MiB
       capabilities: pci apm upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
  *-memory
       description: System Memory
       physical id: 2e
       slot: System board or motherboard
       size: 64GiB
     *-bank:0
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: CMK32GX4M2B3000C15
          vendor: Corsair
          physical id: 0
          serial: 00000000
          slot: DIMM_A1
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
     *-bank:1
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: CMK32GX4M2B3000C15
          vendor: Corsair
          physical id: 1
          serial: 00000000
          slot: DIMM_A2
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
     *-bank:2
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: CMK32GX4M2B3000C15
          vendor: Corsair
          physical id: 2
          serial: 00000000
          slot: DIMM_B1
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
     *-bank:3
          description: DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
          product: CMK32GX4M2B3000C15
          vendor: Corsair
          physical id: 3
          serial: 00000000
          slot: DIMM_B2
          size: 16GiB
          width: 64 bits
          clock: 2133MHz (0.5ns)
  *-cache:0
       description: L1 cache
       physical id: 30
       slot: L1 - Cache
       size: 384KiB
       capacity: 384KiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=1
  *-cache:1
       description: L2 cache
       physical id: 31
       slot: L2 - Cache
       size: 2MiB
       capacity: 2MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=2
  *-cache:2
       description: L3 cache
       physical id: 32
       slot: L3 - Cache
       size: 4MiB
       capacity: 4MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=3
root@deepsky:~# ls -alh /boot
total 127M
drwxr-xr-x  3 root root 4.0K Oct 19 08:27 .
drwxr-xr-x 25 root root 4.0K Sep  9 13:58 ..
-rw-r--r--  1 root root 233K Sep 10 03:12 config-5.4.0-48-generic
-rw-r--r--  1 root root 233K Oct  5 06:34 config-5.4.0-51-generic
drwxr-xr-x  4 root root 4.0K Oct 15 06:28 grub
lrwxrwxrwx  1 root root   27 Oct 14 06:25 initrd.img -> initrd.img-5.4.0-51-generic
-rw-r--r--  1 root root  48M Oct  6 07:53 initrd.img-5.4.0-48-generic
-rw-r--r--  1 root root  48M Oct 14 06:25 initrd.img-5.4.0-51-generic
lrwxrwxrwx  1 root root   27 Oct 14 06:25 initrd.img.old -> initrd.img-5.4.0-48-generic
-rw-r--r--  1 root root 179K Aug 18 03:46 memtest86+.bin
-rw-r--r--  1 root root 181K Aug 18 03:46 memtest86+.elf
-rw-r--r--  1 root root 181K Aug 18 03:46 memtest86+_multiboot.bin
-rw-------  1 root root 4.6M Sep 10 03:12 System.map-5.4.0-48-generic
-rw-------  1 root root 4.6M Oct  5 06:34 System.map-5.4.0-51-generic
lrwxrwxrwx  1 root root   24 Oct 14 06:25 vmlinuz -> vmlinuz-5.4.0-51-generic
-rw-------  1 root root  12M Sep 10 03:36 vmlinuz-5.4.0-48-generic
-rw-------  1 root root  12M Oct  5 06:54 vmlinuz-5.4.0-51-generic
lrwxrwxrwx  1 root root   24 Oct 14 06:25 vmlinuz.old -> vmlinuz-5.4.0-48-generic
root@deepsky:~# exit

% grep -i swap /etc/fstab


swap was on /dev/sda3 during installation


UUID=5c04123d-f760-4e3a-8e04-88ad2671dc38 none   swap     0       0
behindthecurve
  • 77
  • not sure if I understand right. In case you will set your running kernel as default in grub2 https://askubuntu.com/questions/216398/set-older-kernel-as-default-grub-entry – nobody Oct 19 '20 at 16:21
  • In your Software & Updates do you have the Pre-Release (Proposed) enabled under the Developer Options tab? Sometimes you get newer applications and kernels that way that are not fully ready for deployment. – Terrance Oct 19 '20 at 16:32
  • nobody: I can do that, but that doesn't really solve the problem -- I don't want to stay on 48 forever. And the next update will no doubt make 51 be vmlinuz.old and remove 48 completely. I need to prevent that.

    Terrance: no, I do not have pre-release enabled.

    Thanks for the replies.

     – behindthecurve Oct 19 '20 at 18:33
  • Edit your question and show me sudo dmidecode -s bios-version and free -h and sudo lshw -C memory and ls -alh /boot. Start comments to me with @heynnema or I'll miss them. – heynnema Oct 19 '20 at 20:13
  • @heynnema Information included as requested. Thank you for your reply. – behindthecurve Oct 20 '20 at 02:52
  • Sorry, I wasn't paying attention that the 51 kernel is fine. The 52 kernel just got released. Anyway, boot back to the 48 kernel, then run sudo update-initramfs -u -k all then boot back to the 51 kernel. Sometimes the initramfs does not get updated that tells the kernel mounts and stuff that need to be done. – Terrance Oct 20 '20 at 02:59
  • @Terrance No, you don't want to do it that way. Two reasons. If initrd.img is corrupt, the -u flag will only update a corrupted file... always use -c and create a new initrd.img. The "-k all" will update BOTH -48 and -51 initrd.img files, and you don't want to mess with the -48 file, as it's the only thing that still boots. – heynnema Oct 20 '20 at 15:12
  • @behindthecurve Please see my answer. It's incomplete because we need to discuss your swap. In the mean time, the first part of the answer should solve your problem. Report back. – heynnema Oct 20 '20 at 15:20
  • @heynnema It has always worked fine for me with no failures as long as you are running it from a working kernel. – Terrance Oct 20 '20 at 16:05
  • @Terrance Have you seen the many error messages here on AU, regarding (from memory...) "Trouble decoding initramfs file" at boot time? My experience is the -c will fix that, -u won't. – heynnema Oct 20 '20 at 16:14
  • @heynnema The problem I see here is that the boot image was not properly updated when the new kernel was installed. You can do the -c to create a new one if you want. I am not going to argue that point. But, I have not had the -u fail me because it has updated the installed kernel with the proper boot info. I have ran it on many servers that I work on and it has worked fine, and that's 1,000's of servers. – Terrance Oct 20 '20 at 16:17
  • @Terrance re: 1,000's of servers... you've been a very busy guy :-) Cheers. – heynnema Oct 20 '20 at 16:20
  • No worries on not using the -u for the update-initramfs. Glad that the 52 kernel is working for you. I did notice that the time between 51 and 52 was very short, so that makes me wonder if something was wrong with the 51 kernel. – Terrance Oct 21 '20 at 02:39

1 Answers1

0

Partial answer... edits forthcoming...

BIOS

Asus PRIME X-470 Pro

Your BIOS is current at 5603.

/boot/initrd.img-5.4.0-51-generic

This file may be corrupt. The best way to rebuild it is, boot to the -48 kernel and:

sudo update-initramfs -c -k 5.4.0-51-generic

Then reboot to the -51 kernel.

memtest

Ryzen processors are notorious for memory compatibility issues, so...

Go to https://www.memtest86.com/ and download/run their free memtest to test your memory. Get at least one complete pass of all the 4/4 tests to confirm good memory. With 64G RAM, this WILL take many hours to complete.

swap

You have 64G RAM and 60G swap, which is way excessive for swap, unless you're hibernating. We'll discuss some different sizing. Show me grep -i swap /etc/fstab.

Update #1:

It looks like the -51 kernel is broken for you. We removed it and re-ran Software Updater and it installed the -52 kernel, and it's all good now.

heynnema
  • 70,711
  • I appreciate your continued assistance. grep output added to the end of my original post. I'm interested in your response to this, but I've got lots of disk space.

    I rebuilt the initrd.img per your instructions. The md5 checksum changed, but the attempt to boot the 51 kernel failed in the same way. It may be worth mentioning that I get a completely black screen for a long time until BusyBox comes up.

    As to memtest, I completed this a while ago on your advice while we worked on a kernel panic which was resolved by a reinstall of the system. No errors.

     – behindthecurve Oct 20 '20 at 19:43
  • @behindthecurve Did you use the -c option on the command? If so, you might try sudo debsums -s and see if it finds any bad files... like the -51 kernel. If it doesn't, purge all of the -51 stuff and rerun Software Updater. Regarding swap... if you're not hibernating, a 60G swap partition is nuts... even if you have the hard disk space :-) Would you consider a much smaller /swapfile instead? – heynnema Oct 20 '20 at 20:07
  • Yes, I did use the "-c" option on the command. "debsums -s" exited after a minute or so with no messages. By "purge all of the -51 stuff", do you mean the stuff in /boot or are there other things? Do I simply remove them or do I move the -48 files back into their place? Swap: it's on its own partition, so it would be really tough to change it now. Thanks. – behindthecurve Oct 20 '20 at 23:27
  • I followed your suggestion and removed the 51 kernel stuff from /boot, unwarily not looking for anything else to remove. I then ran software update to get the 52 kernel, while leaving the 48 kernel in /boot (after making a copy just in case). I'm happy to report that the 52 kernel boots just fine. I've updated my original post to reflect this. Thank you for your kind assistance. – behindthecurve Oct 21 '20 at 02:16
  • @behindthecurve Great news! I updated my answer. – heynnema Oct 21 '20 at 02:49
  • @behindthecurve re: the swap... no problem... to change over to a smaller /swapfile is easy, and then you can allocate that 60G space back to your Ubuntu partition. Let me know if you want instructions. – heynnema Oct 21 '20 at 02:51