1

Issue

  1. Ubuntu-14.04-LTS (Server) installed.
  2. It worked fine in the past year.
  3. Now, whenever I try to install a package file-system crashes and remounts as read-only.

Working on it

Trying to install a package (via dpkg -i or apt-get install) results in:

EXT4-fs error: mb_free_blocks:1433: group 3, block 130894:freeing already freed block (bit 32590); block bitmap corrupt.

Aborting journal on device vda1-8.

EXT4-fs error: ext4_journal_check_start:56: Detected aborted journal EXT4-fs: Remounting filesystem read-only EXT4-fs: Remounting filesystem read-only EXT4-fs error: ext4_mb_generate_buddy:756: group 3, 8181 clusters in bitmap, 8185 in gd; block bitmap corrupt.

fopen: Read-only file system debconf: DbDriver "config": could not write /var/cache/debconf/config.dat-new: Read-only file system dpkg: error processing package man-db (--install): subprocess installed post-installation script returned error exit status 1 dpkg: error: unable to create new file '/var/lib/dpkg/available-new': Read-only file system

The reason it mounts on read-only mode is because the default setting in /etc/fstab:

UUID=[MY-FS-UUID] / ext4 errors=remount-ro 0 1

In /var/log/syslog I can find:

EXT4-fs: INFO: recovery required on readonly filesystem EXT4-fs: write access will be enabled during recovery EXT4-fs: orphan cleanup on readonly fs EXT4-fs: 3 orphan inodes deleted EXT4-fs: recovery complete EXT4-fs: mounted filesystem with ordered data mode. Opts: (null) Adding 1046524k swap. Priority:-1 extents:1 across:1046524k FS EXT4-fs: re-mounted. Opts: errors=remount-ro

I tried using some of the solutions in the following:

  1. How to fix "sudo: unable to open ... Read-only file system"?

  2. Ubuntu boots in read-only filesystem after upgrade!

by doing so:

  • mount -o remount,rw -t ext4 MY_DISK /

mount: cannot remount block device MY_DISK read-write, is write-protected

  • blockdev -v --setrw MY_DISK

set read-write succeeded.

  • hdparm -r 0 MY_DISK

MY_DISK: setting readonly to 0 (off) readonly = 0 (off)

update 1

I can't access the disk. It seems that the system wrote to a place outside the disk and now it's corrupted. I made a QCOW image of the disk. I'm trying to run it as a VM, and analyze the corrupted image.

I installed libguestfs and I'm trying some analysis tools such as virt-cat, virt-rescue and guestfish.

My main goal now is trying to access and read the system logs.

update 2

In order to access the image, I connected the guest image as a device on the host x86 (credits: http://www.randomhacks.co.uk/how-to-recover-fsck-a-qcow2-file/, https://www.scaleway.com/docs/connect-a-block-device-manually/):

# fsck /dev/nbd0p1
/dev/nbd0p1: clean, 62744/983040 files, 447645/3931904 blocks

partition 1 is the os one. On partitions 2,5 I get errors from fsck - those are the extend and swap partitions, respectively.

# mount /dev/nbd0p1 /mnt/crpt
# cat /mnt/crpt/var/log/*

... currently searching through the logs ...

Help?

Any idea what might cause the system issue?

Alexis Wilke
  • 2,707
orangesomethingorange
  • 411
  • 4
  • 8
  • 19
  • Better reboot in recovery mode and do a fsck. – Jos Mar 29 '17 at 12:06
  • can't reboot the machine in recovery - to do so, I have to edit the grub - but, the fs is read only :( – orangesomethingorange Mar 29 '17 at 12:22
  • Can you boot a Live install e.g. from a USB? – Jos Mar 29 '17 at 12:28
  • yes, and that would give me a root shell... sure... but I already have a root shell now.... I want to know what causes the disk crash... formatting and installing from scratch won't help since I need the info that is already found on this disk... thanks a bunch for your quick reply, though :) – orangesomethingorange Mar 29 '17 at 12:33
  • You would have a root shell that could do a fsck without the disk being mounted. You now have a root shell that can't do a fsck because the disk is mounted (preventing repairs) or is not mounted (preventing access to the fsck executable). – Jos Mar 29 '17 at 12:35
  • I can't access the fs, not from a usb stick anyways because I can't change the grub menu... it seems that the system is writing to a place outside the disk image and so it has corrupted the image.... I've made a qcow copy of the image and I'm trying to run it as a VM – orangesomethingorange Apr 04 '17 at 10:26
  • @Jos I managed to check with fsck, please see update 2 :) and thanks! – orangesomethingorange Apr 04 '17 at 12:22

0 Answers0