Kernel-specific failure to mount OS-containing drive

Question

I recently received a Dell Precision 7820 with Ubuntu 20.04 pre-installed. After fully updating (and all was working), I upgraded to 22.04, but unfortunately, this has done something to prevent the OS-containing SSD from being visible on boot (but only sometimes...).

On booting via default settings, I'm greeted with:

ALERT! UUID=51139eac-d829-4030-8044-d1328bdfd2d3 does not exist. Dropping to a shell.

After which, I drop into an initramfs shell.

The punchline (which I arrived at after trying many many things) is this is kernel-dependent.

After enabling legacy boot via ROM in the BIOS (which I share only in case this matters) and ensuring that SATA is configured to use ACHI instead of RAID (as discussed in this question) I'm left with the following situation.

On boot and hitting ESC to offer up the Ubuntu GRUB options (I've now edited the GRUB info to show this by default, but including this so someone can reproduce what I did without that):

Ubuntu
Advanced options for Ubuntu
UEFI Firmware Settings 
Restore OS to factory state

If I select Advanced options for Ubuntu and then boot into the 5.19.0-46 kernel (which is the default if I just select Ubuntu from the GRUB list), then this fails to mount and drop into initramfs.

HOWEVER, if I boot into the 5.15.0-76 kernel, Ubuntu starts fine with no issue.

Any help to explain this would be hugely appreciated. I'm seriously concerned that I don't understand why this is happening, which means I'm effectively pinned to 5.15.0-76 because, as of right now later kernels prevent OS loading!

Update in response to oldfred (July 21s)

Boot repair link here

============================== Boot Info Summary ===============================
=> No boot loader is installed in the MBR of /dev/nvme0n1.
 => libparted MBR boot code is installed in the MBR of /dev/sda.
nvme0n1p1: _____________________________________________________________________
File system:       vfat
Boot sector type:  Windows 8/10/11/2012: FAT32
Boot sector info:  No errors found in the Boot Parameter Block.
Operating System:  
Boot files:        /efi/Boot/fbx64.efi /efi/Boot/mmx64.efi 
                   /efi/ubuntu/grubx64.efi /efi/ubuntu/loadefi.efi 
                   /efi/ubuntu/mmx64.efi /efi/ubuntu/shimx64.efi 
                   /efi/ubuntu/UbuntuSecBoot.efi /efi/ubuntu/grub.cfg


nvme0n1p2: _____________________________________________________________________
File system:       vfat
Boot sector type:  Windows 8/10/11/2012: FAT32
Boot sector info:  No errors found in the Boot Parameter Block.
Operating System:  
Boot files:        /boot/grub/grub.cfg


nvme0n1p3: _____________________________________________________________________
File system:       ext4
Boot sector type:  -
Boot sector info: 
Operating System:  Ubuntu 22.04.2 LTS
Boot files:        /boot/grub/grub.cfg /etc/fstab /etc/default/grub


sda1: __________________________________________________________________________
File system:       ext4
Boot sector type:  -
Boot sector info: 
Operating System:  
Boot files:        


sda2: __________________________________________________________________________
File system:       ext4
Boot sector type:  -
Boot sector info: 
Operating System:  
Boot files:        



================================ 1 OS detected =================================
OS#1:   Ubuntu 22.04.2 LTS on nvme0n1p3
================================ Host/Hardware =================================
CPU architecture: 64-bit
Video: GA104GL [RTX A4000] EFI VGA from NVIDIA Corporation
BOOT_IMAGE of the installed session in use:
/boot/vmlinuz-5.15.0-76-generic root=UUID=51139eac-d829-4030-8044-d1328bdfd2d3 ro quiet splash vt.handoff=7
df -Th / : /dev/nvme0n1p3 ext4  1.9T  179G  1.6T  11% /
===================================== UEFI =====================================
BIOS/UEFI firmware: 2.32.1(2.32) from Dell Inc.
The firmware is EFI-compatible, and is set in EFI-mode for this installed-session.
SecureBoot disabled (confirmed by mokutil).
BootCurrent: 0007
Timeout: 1 seconds
BootOrder: 0007,0029,002A,002B,002C,002D,0000,0025,0026,0027,0028,0023
Boot0000* ubuntu_shimx64    NVMe(0x1,01-00-00-00-00-00-00-00)/HD(1,GPT,cf73badc-18d9-4b5f-afe6-5c279ce0d278,0x800,0x1a9000)/File(\EFI\ubuntu\shimx64.efi)
Boot0001* Diskette Drive    BBS(Floppy,Diskette Drive,0x0)..BO
Boot0002* Internal HDD  BBS(HD,Internal HDD,0x0)..BO
Boot0003* USB Storage Device    BBS(USB,USB Storage Device,0x0)..BO
Boot0004* CD/DVD/CD-RW Drive    BBS(CDROM,CD/DVD/CD-RW Drive,0x0)..BO
Boot0005* Onboard NIC   BBS(Network,IBA CL Slot 00FE v0110,0x0)..BO
Boot0006* load_efi  HD(1,GPT,cf73badc-18d9-4b5f-afe6-5c279ce0d278,0x800,0x1a9000)/File(\EFI\Ubuntu\loadefi.efi)
Boot0007* ubuntu    HD(1,GPT,cf73badc-18d9-4b5f-afe6-5c279ce0d278,0x800,0x1a9000)/File(\EFI\ubuntu\shimx64.efi)
Boot0008* Onboard NIC(IPV4) PciRoot(0x0)/Pci(0x1f,0x6)/MAC(cc96e52d71d5,0)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0009* Onboard NIC(IPV6) PciRoot(0x0)/Pci(0x1f,0x6)/MAC(cc96e52d71d5,0)/IPv6([::]:<->[::]:,0,0)..BO
Boot000A* PXE IP4 Intel(R) Ethernet 10G 2P X710-T2L-t Adapter   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b48351081138,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000B* PXE IP6 Intel(R) Ethernet 10G 2P X710-T2L-t Adapter   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b48351081138,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot000C* PXE IP4 Intel(R) Ethernet Network Adapter X710-TL PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b48351081139,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000D* PXE IP6 Intel(R) Ethernet Network Adapter X710-TL PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b48351081139,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot0023* UEFI: Hard Drive  HD(1,GPT,cf73badc-18d9-4b5f-afe6-5c279ce0d278,0x800,0x1a9000)/File(EFI\boot\bootx64.efi)..BO
Boot0025  PXE IP4 Intel(R) Ethernet 10G 2P X710-T2L-t Adapter   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b48351081138,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0026  PXE IP6 Intel(R) Ethernet 10G 2P X710-T2L-t Adapter   PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b48351081138,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot0027  PXE IP4 Intel(R) Ethernet Network Adapter X710-TL PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b48351081139,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0028  PXE IP6 Intel(R) Ethernet Network Adapter X710-TL PciRoot(0x1)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b48351081139,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot0029* Diskette Drive    BBS(Floppy,Diskette Drive,0x0)..BO
Boot002A* HGST HUS726T4TALA6L0  BBS(HD,P0: HGST HUS726T4TALA6L0,0x0)..BO
Boot002B* USB Storage Device    BBS(USB,USB Storage Device,0x0)..BO
Boot002C* CD/DVD/CD-RW Drive    BBS(CDROM,CD/DVD/CD-RW Drive,0x0)..BO
Boot002D  Onboard NIC   BBS(Network,IBA CL Slot 00FE v0110,0x0)..BO
============================= Drive/Partition Info =============================
Disks info: ____________________________________________________________________
nvme0n1 : is-GPT,   no-BIOSboot,    has---ESP,  not-usb,    not-mmc, has-os,    no-wind,    2048 sectors * 512 bytes
sda : notGPT,   no-BIOSboot,    has-noESP,  not-usb,    not-mmc, no-os, no-wind,    2048 sectors * 512 bytes
Partitions info (1/3): _________________________________________________________
nvme0n1p3   : is-os,    64, apt-get,    signed grub-efi ,   grub2,  grub-install,   grubenv-ok, update-grub,    farbios
nvme0n1p1   : no-os,    64, nopakmgr,   no-docgrub, nogrub, nogrubinstall,  no-grubenv, noupdategrub,   not-far
sda2    : no-os,    64, nopakmgr,   no-docgrub, nogrub, nogrubinstall,  no-grubenv, noupdategrub,   farbios
sda1    : no-os,    64, nopakmgr,   no-docgrub, nogrub, nogrubinstall,  no-grubenv, noupdategrub,   farbios
Partitions info (2/3): _________________________________________________________
nvme0n1p3   : isnotESP, fstab-has-goodEFI,  no-nt,  no-winload, no-recov-nor-hid,   no-bmgr,    notwinboot
nvme0n1p1   : is---ESP, part-has-no-fstab,  no-nt,  no-winload, no-recov-nor-hid,   no-bmgr,    notwinboot
sda2    : isnotESP, part-has-no-fstab,  no-nt,  no-winload, no-recov-nor-hid,   no-bmgr,    notwinboot
sda1    : isnotESP, part-has-no-fstab,  no-nt,  no-winload, no-recov-nor-hid,   no-bmgr,    notwinboot
Partitions info (3/3): _________________________________________________________
nvme0n1p3   : not--sepboot, with-boot,  fstab-without-boot, not-sep-usr,    with--usr,  fstab-without-usr,  std-grub.d, nvme0n1
nvme0n1p1   : not--sepboot, no---boot,  part-has-no-fstab,  not-sep-usr,    no---usr,   part-has-no-fstab,  no--grub.d, nvme0n1
sda2    : maybesepboot, no---boot,  part-has-no-fstab,  not-sep-usr,    no---usr,   part-has-no-fstab,  no--grub.d, sda
sda1    : maybesepboot, no---boot,  part-has-no-fstab,  not-sep-usr,    no---usr,   part-has-no-fstab,  no--grub.d, sda
fdisk -l (filtered): ___________________________________________________________
Disk nvme0n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk identifier: C926DE13-294F-4E1C-B428-0EFB34E21068
             Start        End    Sectors  Size Type
nvme0n1p1     2048    1742847    1740800  850M EFI System
nvme0n1p2  1742848   18520063   16777216    8G Microsoft reserved
nvme0n1p3 18520064 4000796671 3982276608  1.9T Linux filesystem
Disk sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk identifier: 0x5e6432b3
      Boot      Start        End    Sectors  Size Id Type
sda1             2048 4032045055 4032043008  1.9T 83 Linux
sda2       4032045056 7814035455 3781990400  1.8T 83 Linux
parted -lm (filtered): _________________________________________________________
sda:4001GB:scsi:512:512:msdos:ATA HGST HUS726T4TAL:;
1:1049kB:2064GB:2064GB:ext4::;
2:2064GB:4001GB:1936GB:ext4::;
nvme0n1:2048GB:nvme:512:512:gpt:PC801 NVMe SK hynix 2TB:;
1:1049kB:892MB:891MB:fat32:EFI system partition:boot, esp;
2:892MB:9482MB:8590MB:fat32:Basic data partition:msftres;
3:9482MB:2048GB:2039GB:ext4::;
blkid (filtered): ______________________________________________________________
NAME        FSTYPE   UUID                                 PARTUUID                             LABEL  PARTLABEL
sda

├─sda1      ext4     d06f911f-9f8f-4218-9e4a-c0327d518b77 5e6432b3-01

└─sda2      ext4     fee50400-d622-4de9-aded-759a562617f5 5e6432b3-02

sdb

nvme0n1

├─nvme0n1p1 vfat     94F3-F0B1                            cf73badc-18d9-4b5f-afe6-5c279ce0d278 ESP    EFI system partition
├─nvme0n1p2 vfat     8819-913C                            b3c175c3-e3a0-4ef3-bea5-e11c2b133e84 OS     Basic data partition
└─nvme0n1p3 ext4     51139eac-d829-4030-8044-d1328bdfd2d3 fa4918ab-47f2-4c7b-9eb5-de5999d37c73 UBUNTU
Mount points (filtered): _______________________________________________________
                                           Avail Use% Mounted on

/dev/nvme0n1p3                                  1.6T  10% /
/dev/sda1                                       1.8T   0% /mnt/boot-sav/sda1
/dev/sda2                                       1.6T   0% /mnt/boot-sav/sda2
//storage1.ris.wustl.edu/alex.holehouse/Active  2.8T  89% /work
Mount options (filtered): ______________________________________________________
=================== nvme0n1p1/efi/ubuntu/grub.cfg (filtered) ===================
search.fs_uuid 51139eac-d829-4030-8044-d1328bdfd2d3 root 
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg
=================== nvme0n1p2/boot/grub/grub.cfg (filtered) ====================
Install Complete, remove media and reboot.
Dell Recovery
================= nvme0n1p2: Location of files loaded by Grub ==================
       GiB - GB             File                                 Fragment(s)
        ?? = ??             boot/grub/grub.cfg                             1


=================== nvme0n1p3/boot/grub/grub.cfg (filtered) ====================
Ubuntu   51139eac-d829-4030-8044-d1328bdfd2d3
Ubuntu, with Linux 5.19.0-46-generic   51139eac-d829-4030-8044-d1328bdfd2d3
Ubuntu, with Linux 5.15.0-76-generic   51139eac-d829-4030-8044-d1328bdfd2d3
END /etc/grub.d/30_os-prober
UEFI Firmware Settings   uefi-firmware
END /etc/grub.d/30_uefi-firmware
Restore OS to factory state
======================== nvme0n1p3/etc/fstab (filtered) ========================
<file system> <mount point>   <type>  <options>       <dump>  <pass>
/ was on /dev/nvme0n1p3 during installation
UUID=51139eac-d829-4030-8044-d1328bdfd2d3 /               ext4    errors=remount-ro 0       1
/boot/efi was on /dev/nvme0n1p1 during installation
UUID=94F3-F0B1  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0
==================== nvme0n1p3/etc/default/grub (filtered) =====================
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=lsb_release -i -s 2&gt; /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""
================= nvme0n1p3: Location of files loaded by Grub ==================
       GiB - GB             File                                 Fragment(s)

1738.973667145 = 1867.208757248 boot/grub/grub.cfg                             1
 158.686183929 = 170.387992576  boot/vmlinuz                                   1
 477.052783966 = 512.231526400  boot/vmlinuz-5.15.0-76-generic                 2
 158.686183929 = 170.387992576  boot/vmlinuz-5.19.0-46-generic                 1
 477.052783966 = 512.231526400  boot/vmlinuz.old                               2
 253.857051849 = 272.576933888  boot/initrd.img                                1
1677.781299591 = 1801.503952896 boot/initrd.img-5.15.0-76-generic              1
 253.857051849 = 272.576933888  boot/initrd.img-5.19.0-46-generic              1
1677.781299591 = 1801.503952896 boot/initrd.img.old                            1
=================== nvme0n1p3: ls -l /etc/grub.d/ (filtered) ===================
-rwxr-xr-x 1 root root 18683 Dec 18  2022 10_linux
-rwxr-xr-x 1 root root 43031 Dec 18  2022 10_linux_zfs
-rwxr-xr-x 1 root root 14387 Dec 18  2022 20_linux_xen
-rwxr-xr-x 1 root root 13369 Dec 18  2022 30_os-prober
-rwxr-xr-x 1 root root  1372 Dec 18  2022 30_uefi-firmware
-rwxr-xr-x 1 root root   700 Feb 20  2022 35_fwupd
-rwxr-xr-x 1 root root   214 Apr 15  2020 40_custom
-rwxr-xr-x 1 root root   215 Dec 18  2022 41_custom
-rwxr-xr-x 1 root root  1362 Jul 13 06:23 99_dell_recovery
==================== nvme0n1p3/etc/grub.d/99_dell_recovery =====================
#!/bin/bash -e
source /usr/lib/grub/grub-mkconfig_lib
cat << EOF
menuentry "Restore OS to factory state" {
        search --no-floppy --hint '(hd0,gpt2)' --set --fs-uuid 8819-913C
        set uuid_options="uuid=8819-913C"
        if [ -s /factory/common.cfg ]; then
            source /factory/common.cfg
        else
            set options="boot=casper automatic-ubiquity noprompt quiet splash nomodeset nopersistent"
        fi
        if [ -s /factory/post-rts-gfx.cfg ]; then
            source /factory/post-rts-gfx.cfg
        fi
        if [ -s /factory/post-rts-wlan.cfg ]; then
            source /factory/post-rts-wlan.cfg
        fi
        #Support starting from a loopback mount (Only support ubuntu.iso for filename)
        if [ -f /ubuntu.iso ]; then
            loopback loop /ubuntu.iso
            set root=(loop)
            set options="iso-scan/filename=/ubuntu.iso $options"
        fi
        if [ -n "${lang}" ]; then
            set options="locale=$lang $options"
        fi
        if [ -s /factory/dual_enable ]; then
            set options="dell-recovery/dual_boot=true $options"
        fi
        kernel=/casper/vmlinuz
        if [ ! -f $kernel ]; then
            kernel=/casper/vmlinuz.efi
        fi
        linux   $kernel dell-recovery/recovery_type=hdd $uuid_options $options
        initrd  /casper/initrd
}
EOF
Suggested repair: ______________________________________________________________
The default repair of the Boot-Repair utility would reinstall the grub-efi of
nvme0n1p3,
using the following options:  nvme0n1p1/boot/efi
Additional repair would be performed: unhide-bootmenu-10s use-standard-efi-file
Final advice in case of suggested repair: ______________________________________
Please do not forget to make your UEFI firmware boot on the Ubuntu 22.04.2 LTS entry (nvme0n1p1/efi/**/grub.efi (** will be updated in the final message) file) !

You show 4TB drive as DOS? Drives over 2TB must be gpt or they in past used some proprietary work around.

The 4 TB disk is partitioned into two separate partitions each under 2 TBs. No booting should be coming from here (and on a Live USB this disk is automatically mounted without issue when the Live USB starts). Output from gdisk below

sudo gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.8
Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present

Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

Disk /dev/sda: 7814037168 sectors, 3.6 TiB
Model: HGST HUS726T4TAL
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 6FE8D04B-19DD-4F28-B8E0-6D9A12D769EA
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3693 sectors (1.8 MiB)
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      4032045055   1.9 TiB     8300  Linux filesystem
   2      4032045056      7814035455   1.8 TiB     8300  Linux filesystem

Update in response to oldfred (July 24th)

Best to use UUID, not device as that can change.

Agreed, I have changed this back.

But your device entry does not look correct. You do not have p33 or thirty three partitions on NVMe drive.

Correct, sorry this was a typo in my answer which I've fixed

Root partition should also be 1 for pass parameter, so ext4 fsck can be run on boot.

I'm a confused by this comment; where do you see the pass parameter being not set to 1? In /etc/fstab the root partition has the pass parameter set to 1:

UUID=51139eac-d829-4030-8044-d1328bdfd2d3 / ext4 errors=remount-ro 0 1

But maybe I'm missing something elsewhere?

Its drive size over 2TB must be gpt, not each partition. Not sure why you were even able to create that. But many advantages to gpt anyway.

To simply things I've deleted the sda partitions, such that the entire sda device is unallocated. This does not seem to change any of the behavior discussed above, but I figured it might rule out possible things. For completeness, the Boot-Repair output after this is linked here - I realize pasting would be better but the answer body becomes too long if pasted.

What video driver? And is it correctly installed from Ubuntu repository? Old install may have had it, update would not have automatically reinstalled it, and recovery mode uses nomodeset for video.

Using nvidia drivers that in the 5.15.0-76 kernel work absolutely fine (CUDA accessible, nvidia-smi works etc). Drivers are:

NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |

One thing to note: booting with debug added to the boot line shows that the difference between 5.15.0-76 and 5.19.0-46 is at the point where the boot drive is loaded. Bascically for 5.19 it hangs at

async_tx api initialized (async)

whereas for 5.15 this progress forward to:

EXT4-fs (nvme0n1p3): mounted file system

and then on.

I have messed around a lot with this, including setting rootdelay= to 60 to see if giving it more time will help, but no cigar.

Extra context that brought me here

It took me a lot of hours to get here, so I wanted to share some of the things I did

Initially checked that the 51139eac-d829-4030-8044-d1328bdfd2d3 device was not corrupted. I could boot via a live USB and mount the offending disk just fine. The disk is at /dev/nvme0n1p and the specific Ubuntu OS partition at /dev/nvme0n1p3, as expected for an SSD connected via NVME.

Output of blkid (run after mounting the from the Live USB)

$ blkid
/dev/nvme0n1p1: LABEL="ESP" UUID="94F3-F0B1" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="cf73badc-18d9-4b5f-afe6-5c279ce0d278"
/dev/nvme0n1p2: LABEL="OS" UUID="8819-913C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="Basic data partition" PARTUUID="b3c175c3-e3a0-4ef3-bea5-e11c2b133e84"
/dev/nvme0n1p3: LABEL="UBUNTU" UUID="51139eac-d829-4030-8044-d1328bdfd2d3" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="fa4918ab-47f2-4c7b-9eb5-de5999d37c73"
/dev/sda1: UUID="d06f911f-9f8f-4218-9e4a-c0327d518b77" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5e6432b3-01"
/dev/sda2: UUID="fee50400-d622-4de9-aded-759a562617f5" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5e6432b3-02"
/dev/sdc1: BLOCK_SIZE="2048" UUID="2023-02-23-04-13-44-00" LABEL="Ubuntu 22.04.2 LTS amd64" TYPE="iso9660" PARTLABEL="ISO9660" PARTUUID="a0891d7e-b930-4513-94d8-f629dbd637b2"
/dev/loop1: TYPE="squashfs"
/dev/loop8: TYPE="squashfs"
/dev/loop6: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"
/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="ESP" LABEL="ESP" UUID="F7DB-4D56" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="Appended2" PARTUUID="a0891d7e-b930-4513-94db-f629dbd637b2"
/dev/sdc4: LABEL="writable" UUID="168e1b81-4522-456d-955e-ffae6ecac2c8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="a0829cd9-4a56-af45-87a1-0dc8d8fdd0ff"
/dev/loop7: TYPE="squashfs"
/dev/loop5: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"

and the output from fdisk is:

sudo fdisk -l
Disk /dev/loop0: 2.54 GiB, 2731876352 bytes, 5335696 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
... skipping loop1-loop7 for brevity
Disk /dev/nvme0n1: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk model: PC801 NVMe SK hynix 2TB

Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C926DE13-294F-4E1C-B428-0EFB34E21068
Device            Start        End    Sectors  Size Type
/dev/nvme0n1p1     2048    1742847    1740800  850M EFI System
/dev/nvme0n1p2  1742848   18520063   16777216    8G Microsoft reserved
/dev/nvme0n1p3 18520064 4000796671 3982276608  1.9T Linux filesystem
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: HGST HUS726T4TAL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5e6432b3
Device     Boot      Start        End    Sectors  Size Id Type
/dev/sda1             2048 4032045055 4032043008  1.9T 83 Linux
/dev/sda2       4032045056 7814035455 3781990400  1.8T 83 Linux
Disk /dev/sdc: 7.5 GiB, 8053063680 bytes, 15728640 sectors
Disk model: Flash Disk

Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A0891D7E-B930-4513-94D9-F629DBD637B2
Device       Start      End Sectors  Size Type
/dev/sdc1       64  9613459 9613396  4.6G Microsoft basic data
/dev/sdc2  9613460  9623527   10068  4.9M EFI System
/dev/sdc3  9623528  9624127     600  300K Microsoft basic data
/dev/sdc4  9625600 15728576 6102977  2.9G Linux filesystem
Disk /dev/loop8: 346.33 MiB, 363151360 bytes, 709280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

And the content of fstab on the OS disk is

# note mount point defined from the live USB
cat /mnt/mydisk/etc/fstab
/etc/fstab: static file system information.

Use 'blkid' to print the universally unique identifier for a
device; this may be used with UUID= as a more robust way to name devices
that works even if disks are added and removed. See fstab(5).

<file system> <mount point>   <type>  <options>       <dump>  <pass>
/ was on /dev/nvme0n1p3 during installation
UUID=51139eac-d829-4030-8044-d1328bdfd2d3 /               ext4    errors=remount-ro 0       1
/boot/efi was on /dev/nvme0n1p1 during installation
UUID=94F3-F0B1  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0
#UUID=d06f911f-9f8f-4218-9e4a-c0327d518b77 /mnt/local1  ext4 defaults  0  0
#UUID=fee50400-d622-4de9-aded-759a562617f5 /mnt/local2  ext4 defaults  0  0

My boot mode is set to UEFI: Secure Boot Off, and from the BIOS I had checked the enable Legacy Boot ROM to load the USB mounter.

I also tried updating fstab to mount via the actual path (i.e. changing

UUID=51139eac-d829-4030-8044-d1328bdfd2d3 / ext4 errors=remount-ro 0 1

To

/dev/nvme0n1p3 / ext4 errors=remount-ro 0 1 (NB this previously was written as nvme0n1p33 which was a typo on the answer)

Which also does not make a difference - loading in 5.15.0.76 works either way, and loading with 5.19.0.76 fails either way.