LVM volume not accesible after accidental drive removal, now LV Status is suspended

Question

I have a home server with a pool of 3 drives, and the OS installed in a thumb drive, this thumb drive had physical errors and I had to reinstall the OS, the LVM pool was working well after thumb drive crash, but I accidentally selected one of the drives to manually install grub and things went wrong.

I don't know how/why this drive got detached from the pool and LV stopped working. I've trying for months and have followed different guides and instructions, I partially restored the pool, but it is still not working.

The only clue that I have now for the reason of the error is:

lvdisplay -m
  --- Logical volume ---
  LV Path                /dev/DATA/shared
  LV Name                shared
  VG Name                DATA
  LV UUID                WfdvYe-pga2-dyUn-d2gd-L0xS-KBee-x2PPzP
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              suspended
  # open                 0
  LV Size                6.37 TiB
  Current LE             1669260
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0

  --- Segments ---
  Logical extent 0 to 476931:
    Type                linear
    Physical volume     /dev/sdb1
    Physical extents    0 to 476931

  Logical extent 476932 to 953863:
    Type                linear
    Physical volume     /dev/sda1
    Physical extents    0 to 476931

  Logical extent 953864 to 1669259:
    Type                linear
    Physical volume     /dev/sdc
    Physical extents    0 to 715395

  pvdisplay
   --- Physical volume ---
    PV Name               /dev/sdb1
    VG Name               DATA
    PV Size               1.82 TiB / not usable 1.09 MiB
    Allocatable           yes (but full)
    PE Size               4.00 MiB
    Total PE              476932
    Free PE               0
    Allocated PE          476932
    PV UUID               kcSzc0-tGcn-gA33-w27c-J4MP-Jekh-btQdKr
    --- Physical volume ---
    PV Name               /dev/sda1
    VG Name               DATA
    PV Size               1.82 TiB / not usable 1.09 MiB
    Allocatable           yes (but full)
    PE Size               4.00 MiB
    Total PE              476932
    Free PE               0
    Allocated PE          476932
    PV UUID               0gZJv2-iRwc-r5Z3-VPEb-HUzc-eyAO-LIu2zt
    --- Physical volume ---
    PV Name               /dev/sdc
    VG Name               DATA
    PV Size               2.73 TiB / not usable 472.00 KiB
    Allocatable           yes
    PE Size               4.00 MiB
    Total PE              715397
    Free PE               1
    Allocated PE          715396
    PV UUID               YicF5n-Xb8y-f3GL-kFuM-stxZ-gpmv-0E8mie
    "/dev/sdd5" is a new physical volume of "1.82 TiB"
    --- NEW Physical volume ---
    PV Name               /dev/sdd5
    VG Name
    PV Size               1.82 TiB
    Allocatable           NO
    PE Size               0
    Total PE              0
    Free PE               0
    Allocated PE          0
    PV UUID               LmJ6VD-LHS5-0BVb-mqzB-gbbc-8dRX-M74APj

Note: The 4th drive sdd5 was never part of the LVM DATA pool, and have no data.

sudo fdisk -l

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd53b391a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  3907029167  1953514583+  8e  Linux LVM

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x6b4f5902

   Device Boot      Start         End      Blocks   Id  System

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00081354

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048      499711      248832   83  Linux
/dev/sdd2          501758  3907028991  1953263617    5  Extended
Partition 2 does not start on physical sector boundary.
/dev/sdd5          501760  3907028991  1953263616   8e  Linux LVM

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x19e4252c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3902855167  1951426560   8e  Linux LVM
/dev/sdb2      3902857214  3907028991     2085889    5  Extended
/dev/sdb5      3902857216  3907028991     2085888   8e  Linux LVM

Disk /dev/sde: 16.1 GB, 16097738752 bytes
64 heads, 32 sectors/track, 15352 cylinders, total 31440896 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cad44

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2046     6125567     3061761    5  Extended
/dev/sde2   *     6125568    31438847    12656640   83  Linux
/dev/sde5            2048     1953791      975872   83  Linux
/dev/sde6         1955840     6125567     2084864   82  Linux swap / Solaris

*Note: The 16GB drive is the thumbdrive where OS is installed

updated result of fdisk -lon Jan-20th after execute @psusi suggestions

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd53b391a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  3907029167  1953514583+  8e  Linux LVM

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x19e4252c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907028991  1953513472   83  Linux

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x6b4f5902

   Device Boot      Start         End      Blocks   Id  System

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00081354

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048      499711      248832   83  Linux
/dev/sdd2          501758  3907028991  1953263617    5  Extended
Partition 2 does not start on physical sector boundary.
/dev/sdd5          501760  3907028991  1953263616   8e  Linux LVM

Disk /dev/sde: 16.1 GB, 16097738752 bytes
64 heads, 32 sectors/track, 15352 cylinders, total 31440896 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cad44

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2046     6125567     3061761    5  Extended
/dev/sde2   *     6125568    31438847    12656640   83  Linux
/dev/sde5            2048     1953791      975872   83  Linux
/dev/sde6         1955840     6125567     2084864   82  Linux swap / Solaris

Please add a -m to lvdisplay and update. Also add pvdisplay — psusi, Dec 23 '13 at 20:33
I have added the info you requested. Thank you for your help! — hirofairlane, Dec 23 '13 at 21:07
To make sure I see your comment, prefix it with an @psusi. Lucky I happened to come back and check today. Which drive is detached? It looks like everything is working there, though that logical volume is spanned across all three drives, which is not good from a data safety perspective: if any one of them fails, the whole volume is lost. Also you might try lvchange -ay DATA/shared. — psusi, Dec 27 '13 at 14:35
@psusi thank you for your reply, that's the result:
root@shared:~# lvchange -ay DATA/shared device-mapper: resume ioctl on failed: Invalid argument Unable to resume DATA-shared (252:0) — hirofairlane, Jan 05 '14 at 10:04
Strange, is there anything relevant in the output of dmesg? — psusi, Jan 05 '14 at 18:14
that's what I get when I try lvchange -ay DATA/shared.... .device-mapper: table: 252:0: sdb1 too small for target: start=384, len=3907026944, dev_size=3902853120.
@psusi Sorry I wasn't able to try it before. — hirofairlane, Jan 10 '14 at 18:22
@psusi Thank you for your help, I have added the output of fdisk -l — hirofairlane, Jan 19 '14 at 18:41
Hi @psusi that's what I get with a dmesg: [ 1566.773561] bio: create slab <bio-1> at 1 [ 1566.773918] device-mapper: table: 252:0: sdb1 too small for target: start=384, len=3907026944, dev_size=3907026944 — hirofairlane, Jan 21 '14 at 18:25
Hi @psusi one question, I have realized that sdc has no partitions could this be the reason? — hirofairlane, Jan 21 '14 at 19:34
No, the problem is that somehow, sdb is not as large as it is supposed to be. Even after fixing the partition table to maximize the size of the partition, you are still short 176 sectors. I'd say it's time to restore from backup... and check the SMART health of sdb; it may be failing... — psusi, Jan 21 '14 at 19:47
@psusi COuld this be due to starting partition point? it is starting at 2048 instead of starting at 1 like sda1 — hirofairlane, Jan 21 '14 at 20:56
Starting at sector 1 is very goofy ( no partitioners will do that normally ), but if it had originally been there and changed, lvm would no longer recognize the pv... unless you did something really goofy like use pvcreate --uuid to recreate the pv label at the new 2048 sector offset, then restored the previous metadata? — psusi, Jan 21 '14 at 21:13
I guess that this could be the explanation, since I have originally created the pool with sda and sdb, and sda still starts at #1 sector. Could you please help me re-creating the partition at 1? — hirofairlane, Jan 21 '14 at 21:16
It won't work since the lvm super block is at sector 2048, not 1. — psusi, Jan 22 '14 at 00:07
I guess that it's time for me to assume that all my data is lost. Thank you @psusi for all your help. — hirofairlane, Jan 22 '14 at 08:04

score 1 · Answer 1 · answered Jan 19 '14 at 22:05

1

You seem to have clobbered your partition table on sdb. According to lvm, it originally had a single partition occupying the whole disk, and now sdb1 is too small, and much of the space is now unallocated, or allocated to another partition that says it is used by lvm, but lvm says it isn't. You should be able to use parted to blow away all of the partitions on the disk, and recreate the single whole disk partition to repair this.

sudo parted /dev/sdb
rm 5
rm 2
rm 1
mkpart primary ext2 2048s 100%
quit

answered Jan 19 '14 at 22:05

psusi

37,551

I have tried the command as you state, but nothing happened. I have executed a "mount -a" to refresh fstab and the "fdisk -l" remains the same. I suppose that the volume group should be blocking the disk to be partitioned. I'm also curious about how did you notice that sdb started at beginning, and the new partition as ext2. Shouldn't it be in LVM? – hirofairlane Jan 20 '14 at 18:48
@hirofairlane, you will need to reboot for the changes to take affect ( you should have seen a warning message from parted telling you this ). As for ext2, the argument is really meaningless these days and parted doesn't understand a filesystem type for lvm. – psusi Jan 20 '14 at 18:53
I have rebooted the system, but everything is as it was... same outputs in pvdisplay, lvdisplay, and lvchange -ay DATA/shared – hirofairlane Jan 20 '14 at 19:15
@hirofairlane, and fdisk -l? – psusi Jan 20 '14 at 20:04
I have added a reply with the fdisk -l output – hirofairlane Jan 20 '14 at 21:32

LVM volume not accesible after accidental drive removal, now LV Status is suspended

1 Answers1