Repairing, or mounting an encrypted filesystem

Question

I am trying to save data from a system that used to live on a tiny usb stick, but I suspect a hardware failure. The system came out of a suspend with the root filesystem in read only, and instead of understanding that it was about to die and that I should try to save some data, I opted for a quick reboot, but was not able since then to get back into it in any way.

It is an Ubuntu 18.04 with encrypted root filesystem chosen at the time of its installation, so I suppose that is LUKS. If I try to boot into it now, I get the violet screen, then an error message, then I get the (initramfs) prompt. I've tried to do a manual fsck of the filesystem as per the instructions here, but keep getting i/o errors of the following kind,

Buffer I/O error on dev dm-1, logical block 4, lost async page page write

and

print_req_error: critical target error, dev sda, sector 9896192 flags 801.

It ends with

fsck.ext4: unable to set superblock flags on /dev/mapper/ubuntu--vg-root,

And telling me that the filesystem still has errors. I tried specifying backup superblocks, but without success.

I now tried to put the stick into a healthy system and decrypt and mount it and see if I can still save something. When inserting the stick I am prompted for the key, and apparently it decrypts it, see the syslog:

Dec 21 03:38:42 X1 kernel: [36919.500937] usb 2-1: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
Dec 21 03:38:42 X1 kernel: [36919.521737] usb 2-1: New USB device found, idVendor=0781, idProduct=5583, bcdDevice= 1.00
Dec 21 03:38:42 X1 kernel: [36919.521742] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Dec 21 03:38:42 X1 kernel: [36919.521745] usb 2-1: Product: Ultra Fit
Dec 21 03:38:42 X1 kernel: [36919.521748] usb 2-1: Manufacturer: SanDisk
Dec 21 03:38:42 X1 kernel: [36919.521751] usb 2-1: SerialNumber: 4C530001210812112290
Dec 21 03:38:42 X1 kernel: [36919.523118] usb-storage 2-1:1.0: USB Mass Storage device detected
Dec 21 03:38:42 X1 kernel: [36919.523697] scsi host1: usb-storage 2-1:1.0
Dec 21 03:38:42 X1 mtp-probe: checking bus 2, device 3: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-1"
Dec 21 03:38:42 X1 mtp-probe: bus: 2, device: 3 was not an MTP device
Dec 21 03:38:42 X1 upowerd[1354]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0
Dec 21 03:38:42 X1 upowerd[1354]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1
Dec 21 03:38:43 X1 kernel: [36920.553722] scsi 1:0:0:0: Direct-Access     SanDisk  Ultra Fit        1.00 PQ: 0 ANSI: 6
Dec 21 03:38:43 X1 kernel: [36920.554585] sd 1:0:0:0: Attached scsi generic sg1 type 0
Dec 21 03:38:43 X1 kernel: [36920.554754] sd 1:0:0:0: [sdb] 489160704 512-byte logical blocks: (250 GB/233 GiB)
Dec 21 03:38:43 X1 kernel: [36920.555633] sd 1:0:0:0: [sdb] Write Protect is on
Dec 21 03:38:43 X1 kernel: [36920.555641] sd 1:0:0:0: [sdb] Mode Sense: 43 00 80 00
Dec 21 03:38:43 X1 kernel: [36920.555952] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 21 03:38:43 X1 kernel: [36920.566040]  sdb: sdb1 sdb2 < sdb5 >
Dec 21 03:38:43 X1 kernel: [36920.568791] sd 1:0:0:0: [sdb] Attached SCSI removable disk
Dec 21 03:38:43 X1 kernel: [36920.961099] EXT4-fs (sdb1): INFO: recovery required on readonly filesystem
Dec 21 03:38:43 X1 kernel: [36920.961100] EXT4-fs (sdb1): write access unavailable, cannot proceed (try mounting with noload)
Dec 21 03:38:43 X1 gnome-shell[1828]: Unable to mount volume 767 MB Volume: Gio.IOErrorEnum: Error mounting /dev/sdb1 at /media/peter/0b776e27-d039-4fc5-ad20-462e5e36e031: cannot mount /dev/sdb1 read-only
Dec 21 03:38:50 X1 gnome-shell[1828]: Unable to mount volume 250 GB Encrypted: Gio.IOErrorEnum: Operation was cancelled
Dec 21 03:38:50 X1 udisksd[1004]: Unlocked LUKS device /dev/sdb5 as /dev/dm-3

Now

$ sudo fsck /dev/dm-3 -y
fsck from util-linux 2.31.1

Nothing else. Not sure if that's good or bad news, or I am on the wrong track maybe. Just trying to mount it I get unknown filesystem type 'LVM2_member'. The following also doesn't help:

$ udisksctl mount -b /dev/dm-3 
Object /org/freedesktop/UDisks2/block_devices/dm_2d3 is not a mountable filesystem.

Now I still don't really understand how LVM and LUKS work, and I don't know how broken the stick is. What could I try next?

EDIT: I am making another attempt at getting into the disk, inserting it into a healthy system. I get asked for the password, the system is apparently decrypted.

Jan  4 13:15:59 X1 kernel: [258125.811027] usb 2-2: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd
Jan  4 13:15:59 X1 kernel: [258125.835919] usb 2-2: New USB device found, idVendor=0781, idProduct=5583, bcdDevice= 1.00
Jan  4 13:15:59 X1 kernel: [258125.835924] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan  4 13:15:59 X1 kernel: [258125.835927] usb 2-2: Product: Ultra Fit
Jan  4 13:15:59 X1 kernel: [258125.835930] usb 2-2: Manufacturer: SanDisk
Jan  4 13:15:59 X1 kernel: [258125.835933] usb 2-2: SerialNumber: 4C530001210812112290
Jan  4 13:15:59 X1 kernel: [258125.837229] usb-storage 2-2:1.0: USB Mass Storage device detected
Jan  4 13:15:59 X1 kernel: [258125.837738] scsi host1: usb-storage 2-2:1.0
Jan  4 13:15:59 X1 mtp-probe: checking bus 2, device 6: "/sys/devices/pci0000:00/0000:00:14.0/usb2/2-2"
Jan  4 13:15:59 X1 mtp-probe: bus: 2, device: 6 was not an MTP device
Jan  4 13:15:59 X1 upowerd[1314]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb2/2-2/2-2:1.0
Jan  4 13:15:59 X1 upowerd[1314]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:14.0/usb2/2-2
Jan  4 13:16:00 X1 kernel: [258126.856090] scsi 1:0:0:0: Direct-Access     SanDisk  Ultra Fit        1.00 PQ: 0 ANSI: 6
Jan  4 13:16:00 X1 kernel: [258126.856856] sd 1:0:0:0: Attached scsi generic sg1 type 0
Jan  4 13:16:00 X1 kernel: [258126.856963] sd 1:0:0:0: [sdb] 489160704 512-byte logical blocks: (250 GB/233 GiB)
Jan  4 13:16:00 X1 kernel: [258126.857870] sd 1:0:0:0: [sdb] Write Protect is on
Jan  4 13:16:00 X1 kernel: [258126.857875] sd 1:0:0:0: [sdb] Mode Sense: 43 00 80 00
Jan  4 13:16:00 X1 kernel: [258126.858195] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan  4 13:16:00 X1 kernel: [258126.867874]  sdb: sdb1 sdb2 < sdb5 >
Jan  4 13:16:00 X1 kernel: [258126.869975] sd 1:0:0:0: [sdb] Attached SCSI removable disk
Jan  4 13:16:00 X1 kernel: [258127.138645] EXT4-fs (sdb1): INFO: recovery required on readonly filesystem
Jan  4 13:16:00 X1 kernel: [258127.138646] EXT4-fs (sdb1): write access unavailable, cannot proceed (try mounting with noload)
Jan  4 13:16:00 X1 gnome-shell[1852]: Unable to mount volume 767 MB Volume: Gio.IOErrorEnum: Error mounting /dev/sdb1 at /media/peter/0b776e27-d039-4fc5-ad20-462e5e36e031: cannot mount /dev/sdb1 read-only
Jan  4 13:16:03 X1 systemd-resolved[924]: Got packet on unexpected IP range, refusing.
Jan  4 13:16:03 X1 systemd-resolved[924]: Got packet on unexpected IP range, refusing.
Jan  4 13:16:05 X1 gnome-shell[1852]: Unable to mount volume 250 GB Encrypted: Gio.IOErrorEnum: Operation was cancelled
Jan  4 13:16:05 X1 udisksd[1095]: Unlocked LUKS device /dev/sdb5 as /dev/dm-3

If I try to mount /dev/dm-3 I get only.

unknown filesystem type 'LVM2_member'.`

lvmdiskscan shows me the volume:

/dev/mapper/luks-477637dc-44d2-4076-9519-194cc4bf46c6 [ 232,53 GiB] LVM physical volume

But lvs or lvscan don't show me anything related to the usb stick.

I assume it is failing and has gone into a kind of read-only mode. I wonder if there is some way to mount the root partion /dev/sdb5 read-only to still try and get some stuff out from it? Maybe I could dd /dev/dm-3 somewhere else and try to mount it?

score 0 · Answer 1 · answered Feb 16 '20 at 21:07

For the record:

Finally I got to making another try to saving the data on the encrypted partition on the dying USB stick, which apparently had gone into a read-only mode, making it impossible (at least as far as I could see) to mount the actual partition. Inserting the stick into a USB-port still resulted in a password prompt getting displayed, and the partition would be decrypted, but would fail to mount. Thus copying over the decrypted image as a whole, and then mounting it as a loopback device appeared to be the road to take:

sudo dd if=/dev/dm-3 of=tmp/oldstick status=progress

which took about 8 hours for the roughly 250 GB. Next the newly copied image had to be joined with a loopback device.

sudo losetup --all

showed that everything up to /dev/loop17 was already in use, so number 18 has to be taken:

sudo losetup /dev/loop18 tmp/oldstick

After that it appeared in the output of sudo pvs, but with the same VG as the currently running system. A few tries at different invocations of vgrename, then I discovered vgimportclone. Again a few tries failed, the successful one then seems to have been

sudo vgimportclone -i /dev/loop18, after which it appeared in the output of sudo pvs as ubuntu-vg1.

sudo vgchange -a y ubuntu-vg1

would then activate this volume, and sudo lvscan would then, among others, output the following line:

ACTIVE '/dev/ubuntu-vg1/root' [231,57 GiB] inherit

which then pointed me to what device I would actually have to use as argument for the mount command, such that after a few failed attempts

sudo mount -o ro /dev/ubuntu-vg1/root tmp/oldroot/

finally brought me back into my old system.

Inspiration for this approach was mostly taken from here.

Repairing, or mounting an encrypted filesystem

1 Answers1