how to truncate a disk image file of unused space without corrupting GPT partition table (end pointer)

Question

Lots of info on taking a image of a disk and shrinking the root (bionic) filesystem partition and then truncating the image to remove the part that became free space. Like https://softwarebakery.com//shrinking-images-on-linux

So there are basically three steps. Use resize2fs to shrink a filesystem on a partition, then decrease the size of the partition as well. Then finally remove the now unused space in the image file.

It is the third part I am having a problem with. Every time I try dd or truncate all the partitions of the image are removed as though the GPT partition table within the image file was trashed.

Here is the original image

GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk rock64-base.img: 30310400 sectors, 14.5 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 159DCEDE-DBEA-4657-96D9-2CE178A96B7E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 30310366
Partitions will be aligned on 64-sector boundaries
Total free space is 30 sectors (15.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8063   3.9 MiB     8300  loader1
   2            8064            8191   64.0 KiB    8300  reserved1
   3            8192           16383   4.0 MiB     8300  reserved2
   4           16384           24575   4.0 MiB     8300  loader2
   5           24576           32767   4.0 MiB     8300  atf
   6           32768          262143   112.0 MiB   0700  boot
   7          262144        30310366   14.3 GiB    8300  root

Then after first two steps

Found valid GPT with protective MBR; using GPT.
Disk test.img: 30310400 sectors, 14.5 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 159DCEDE-DBEA-4657-96D9-2CE178A96B7E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 30310366
Partitions will be aligned on 64-sector boundaries
Total free space is 23555002 sectors (11.2 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8063   3.9 MiB     8300  loader1
   2            8064            8191   64.0 KiB    8300  reserved1
   3            8192           16383   4.0 MiB     8300  reserved2
   4           16384           24575   4.0 MiB     8300  loader2
   5           24576           32767   4.0 MiB     8300  atf
   6           32768          262143   112.0 MiB   0700  boot
   7          262144         6755394   3.1 GiB     8300  primary

you can see the root filesystem partition has been reduced to 3.1G

I can load that image fine. I can put it back on a SD card and boot my device. So steps 1 and 2 are fine.

NOW...

if I complete the process doing a `truncate --size=$[(6755394+1)*512] 'test.img'

and

GPT fdisk (gdisk) version 1.0.3

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Error 25 reading partition table for CRC check!
Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Disk test2.img: 6755395 sectors, 3.2 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 159DCEDE-DBEA-4657-96D9-2CE178A96B7E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 30310366
Partitions will be aligned on 64-sector boundaries
Total free space is 23555002 sectors (11.2 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8063   3.9 MiB     8300  loader1
   2            8064            8191   64.0 KiB    8300  reserved1
   3            8192           16383   4.0 MiB     8300  reserved2
   4           16384           24575   4.0 MiB     8300  loader2
   5           24576           32767   4.0 MiB     8300  atf
   6           32768          262143   112.0 MiB   0700  boot
   7          262144         6755394   3.1 GiB     8300  primary

Clearly truncate (or dd for that matter) are not playing well with GPT. Running gdisk on this file confirms the gpt is trashed

if I run gdisk on the truncated file it reports a trashed GPT.

So obviously I'm missing something here. Apparently when the end of the disk file changes the GPT gets corrupted or needs to be fixed (mismatch). Even though it is fine/present at the end of step 2. I messed around with gdisk and couldn't repair it. Besides that I wanted a solution I could do with a script and with this GPT it doesn't work.

So because it's a GPT do I need to use something other than truncate or dd or do I have to "fix" the GPT manually after truncating.

Here is what gdisk verify reports

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Problem: The secondary header's self-pointer indicates that it doesn't reside
at the end of the disk. If you've added a disk to a RAID array, use the 'e'
option on the experts' menu to adjust the secondary header's and partition
table's locations.

Problem: Disk is too small to hold all the data!
(Disk size is 6755395 sectors, needs to be 30310400 sectors.)
The 'e' option on the experts' menu may fix this problem.

Problem: GPT claims the disk is larger than it is! (Claimed last usable
sector is 30310366, but backup header is at
30310399 and disk size is 6755395 sectors.
The 'e' option on the experts' menu will probably fix this problem

Partition(s) in the protective MBR are too big for the disk! Creating a
fresh protective or hybrid MBR is recommended.

if the goal is to reduce the size of an image file, simply gzip it ;) — cmak.fr, Nov 18 '19 at 06:00
This image has built in expander on first boot so the goal is to do this so it can be loaded on media of any size big enough to hold the minimum size. I have used for example PiShrink before but on a disk image with MBR. The GPT is causing the issue — DKebler, Nov 18 '19 at 15:43

score 5 · Accepted Answer · answered Nov 18 '19 at 18:13

GPT includes both a main partition table at the start of the disk and a backup partition table at the end of the disk. (Literally the end of the disk -- the final few sectors of the disk.) Thus, when you truncated the disk image, you removed the backup partition table. What's more, some of the pointers and metadata in the main partition table became invalid, since they were pointing past the end of the (virtual) disk. That's most of what the v command in gdisk was complaining about.

None of this means that the disk is completely trashed. So long as the main partition table's data is valid, gdisk (and most other GPT partitioning tools) can recover. In gdisk, you would type x to enter the experts' menu, then type e to move the backup partition table data to the new end of the disk, and then type w to write the changes to disk. (I'd recommend typing v again before w in case I've overlooked another problem; also, see below....)

Note, however, that there is one more problem: Your disk image is 6,755,395 sectors in size, according to gdisk; but the final sector of the last partition is at sector 6,755,394. This doesn't leave enough space for the backup partition table (it consumes 33 sectors, by default). If you can re-do the whole process, you should probably shrink the disk image by a smaller amount. If not, try expanding the disk image by 33*512 (16,896) bytes before re-creating the backup partition table with gdisk.

I can control the truncating as this is a script working of a copy of the original image.
So is there a way to automate this? Can gdisk take arguments for functions to process i.e. 'xew' What I am doing is in a script so would want this "fix" to be part of bash script. — DKebler, Nov 18 '19 at 18:55
Reporting that adding 33 sectors to my truncate command and running gdisk with a x,e,w worked. Albiet the shrunk root partition won't mount (long list of potential errors...). But just realized I can't mount it after first two steps so that is not a GPT issue but rather something in my shrink script. So yes! Rod's advice is good. — DKebler, Nov 18 '19 at 19:20
For use in a script, you should probably use sgdisk, not gdisk. — Rod Smith, Nov 18 '19 at 19:21
Now that I have it working for those who might like a bash script for shrinking and truncating a disk image here is my effort https://gist.github.com/dkebler/92aa919e9aacc8a3f6b6b07c7abe12b4 — DKebler, Nov 18 '19 at 22:01

how to truncate a disk image file of unused space without corrupting GPT partition table (end pointer)

1 Answers1

Linked