20

If I make a disk image and compress it with gzip/xz/etc is there a way to mount it directly without first uncompressing it?

Say I've used 

sudo dd if=/dev/sdc1 | gzip -9 > image1.dd.gz

how can I mount the original image, without creating an uncompressed copy first?

Or I've used

sudo dd if=/dev/sdc | gzip -9 > wholedisk.dd.gz

and the disk has multiple partitions, would that make it any harder?

With an uncompressed image of a whole disk then using kpartx or newer versions of losetup with it's -P flag should create a loop for each partition.

But is there a way to mount/losetup/read the compressed image?

If it won't work for gzip/xz, is there any compression method this would work for?

Note on duplicate Q

The currently suggested duplicate

DOES NOT USE COMPRESSION, and IS NOT A DUPLICATE.

mount will not mount a compressed image by itself.

Xen2050
  • 8,705
  • I can tell you that it definitely is possible. But I can't tell you how atm because the instructions are on 2 external HDD and right now I have access to neither but I want to let you know that it works. I'll come back with an answer when I have one (and no one else posted one until then). – UTF-8 Oct 12 '16 at 20:03
  • 3
    Possible duplicate of Mount single partition from image of entire disk (device) – Panther Oct 12 '16 at 20:20
  • 1
    @UTF-8 That's encouraging, thanks. FYI, original goal is to make a smaller backup of a drive/partition that's nearly empty or has lots of easily compressible data, and be able to read it without writing a LOT of extra data or needing a ton of free space to extract it – Xen2050 Oct 12 '16 at 20:20
  • See also https://major.io/2010/12/14/mounting-a-raw-partition-file-made-with-dd-or-dd_rescue-in-linux/ – Panther Oct 12 '16 at 20:21
  • @bodhi.zazen Thanks, but those don't address the compression, afaik mount & losetup/kpartx don't automatically uncompress images. – Xen2050 Oct 12 '16 at 20:27
  • Then you have answered your own question ;) – Panther Oct 12 '16 at 20:29
  • 2
    @bodhi.zazen Pardon? Maybe I should clarify, mount does not automatically uncompress images, so this Q is not a duplicate of https://askubuntu.com/questions/69363 (maybe I missed the joke, hopefully uncompressing isn't the only answer :) – Xen2050 Oct 12 '16 at 20:40
  • You have answered your own question. "Mount will not mount a compressed image" which is partially true, mount will mount a compressed image if you use squashfs. So you have to decompress it first QED. The question I linked tells you how to mount the partition(s) on the image. – Panther Oct 12 '16 at 20:48
  • @Xen2050 Don't get your hopes up to hear about the solution from me very soon. The earliest point in time I could go where those external HDDs are is in about a week. I'm currently extracting 3 compressed images myself because I need something stored on them and don't want to wait until then. – UTF-8 Oct 12 '16 at 20:49
  • No rush, I'll wait. Maybe there's another creative solution that someone's been hiding, just waiting for a Q like this to answer. Thanks bodhi.zazen, I just found the squashfs method too (from google, the search here on AskUbuntu doesn't extend to other SE sites) and it looks like a good answer too, anyone who gave "just the facts" instructions in an answer here would at least get an upvote from me, maybe a check-mark if there's no other super answers after a while too. – Xen2050 Oct 12 '16 at 20:59
  • Sorry, I probably was mistaken. I can't find any script doing that on my external hard drives. – UTF-8 Oct 22 '16 at 14:52
  • There's an answer over on Unix&Linux that says nbdkit & it's xz plugin can do this with xz compression, but getting it to work (installing / compiling the plugin?) has eluded me. I'd upvote & select correct an answer that showed (plain & simple steps) how to get it to work, especially if it worked with gzip too. – Xen2050 Dec 11 '16 at 09:04
  • see also: https://unix.stackexchange.com/questions/31669/is-it-possible-to-mount-a-gzip-compressed-dd-image-on-the-fly – sdaau Mar 21 '20 at 12:24
  • If you're compressing a whole disk image, -9 seems like a poor choice as it would unnecessarily slow down the compression. -3 is probably a sweet spot where it compresses nearly as well but it'll go much, much faster. – thomasrutter Jul 31 '20 at 04:25

2 Answers2

12

You can use squashfs to compress disk images and then mount them.

Create the disk image

If you haven't got a disk image yet use dd to create one:

dd if=/dev/sda1 of=/tmp/sda1.img bs=4M

Compress the image with squashfs

Install squashfs:

apt-get install squashfs-tools

Compress the image:

mksquashfs /tmp/sda1.img /tmp/sda1.squash.img

Or Stream the compression (don't need a temporary dd file)

mkdir empty-dir
mksquashfs empty-dir squash.img -p 'sda_backup.img f 444 root root dd if=/dev/sda bs=4M'

Compliments to terminator14 at UbuntuForums.org. Definitions/Explanations:

  • empty-dir - "source" dir. Basically in our case, just an empty dir to satisfy mksquashfs' input arg format
  • squash.img - the destination and filename of the output squashfs file
  • sda_backup.img - the name of the dd backup INSIDE the squashfs file
  • f - specifies that sda_backup.img is a regular file (as opposed to a directory, block device, or char device)
  • 444 - permissions of the sda_backup.img file inside the squashfs image
  • root root - UID and GID for the sda_backup.img file inside the squashfs image. Can be specified by decimal numbers, or by name
  • dd if=/dev/sda bs=4M - the dd command used to read the device we want backed up

Mount the image

First mount the squashfs image:

mkdir /mnt/squash
mount /tmp/sda1.squash.img /mnt/squash

This will present the un-compressed disk image for you to mount:

mkdir /mnt/sda1
mount /mnt/squash/sda1.img /mnt/sda1

Or if it's a full drive image (partitioned) you could use losetup to attach the dd image to a loop device (possibly optional) and then kpartx -a or partprobe to find & separate the partitions to separate devices, or even vgscan / vgchange -ay if there's LVM.

Xen2050
  • 8,705
Eborbob
  • 303
  • 1
  • 3
  • 9
  • 2
    I added some on-the-fly squashfs info, to avoid needing a temporary plain dd file. Great answer with squashfs, just doesn't do the more pipe-friendly xz/gz formats – Xen2050 Dec 11 '16 at 11:25
  • Would that somehow work with a full disk image containing multiple partitions as well? – friederbluemle Nov 18 '17 at 17:43
  • See here for full disk: https://unix.stackexchange.com/questions/31669/is-it-possible-to-mount-a-gzip-compressed-dd-image-on-the-fly – HackerBoss Mar 19 '20 at 15:53
  • Depending on the filesystem type, mounting may require the ro,noload options, like so: mount -o ro,noload /mnt/squash/sda1.img /mnt/sda1 – HackerBoss Mar 19 '20 at 16:58
9

I was able to get an NBD server to "serve up" an un-compressed version of a .gz or .xz file. I'm not sure if it has to read the entire compressed file very often (only tested so far with small files in ram) but at least it works without having to extract a whole un-compressed copy.

Viewing a gz / xz Compressed dd image "on-the-fly"

1 - Install nbdkit & client

apt install nbdkit nbd-client

2 (Optional for testing only) - Create demo "dd" style file - an empty 50M file with an ext filesystem & zip it

fallocate -l 50M 50m
mkfs.ext3 -v 50m

Then compress with either gzip or xz

gzip -vk9 50m
xz -v9k 50m

Note: An xz option of --block-size=16MiB should give better random access performance, but at the expense of less compression (maybe 1%?). See the nbdkit's xz plugin description.

3 - Run server on COMPRESSED image

nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-gzip-plugin.so file=~/Downloads/50m.gz

or for xz

nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-xz-plugin.so file=~/Downloads/50m.xz

Apparently newer versions of ndbkit can specify --filter=xz instead of specifying a whole plugin path:

nbdkit -v --no-fork -i 127.0.0.1  --filter=xz file=~/Downloads/50m.xz

  • The --no-fork tells the server "Don't fork into the background" so seeing possible error messages & killing it with CTRL-C is a little easier.

  • -v may / will show too many status messages, especially when reading.

4 - Run client (in another terminal/window)

sudo nbd-client -nofork 127.0.0.1 10809 /dev/nbd0

The -nofork is similar to -no-fork above, but I could not kill the client with CTRL-C, kill <pid>, or sudo kill <pid>. Ending the server with CTRL-C stopped both server & client.

5 - Observe that /dev/nbd0 is now identical to UN-COMPRESSED image

$ sudo diff -s /dev/nbd0 ~/Downloads/50m
Files /dev/nbd0 and /home/user/Downloads/50m are identical

/dev/nbd0 can now be used just like the uncompressed image; mounted, read/copy with dd, or use kpartx, partprobe, vgscan / vgchange, etc!

Thanks to Cristian Ciupitu on Unix & Linux, though it took a little more digging to find the plugin & get everything going together.

Xen2050
  • 8,705
  • On Fedora 31 nbdkit -v --no-fork -i 127.0.0.1 --filter=xz file ~/Downloads/50m.xz worked great with the same client command – EkriirkE Sep 07 '19 at 10:51
  • On Ubuntu xenial 16.04 LTS nbdkit 1.1.11 uses old NBD protocol, which differs from new NBD protocol, used in nbd-client 3.13, but you ca use nbd-client version before 3.10, e.g. nbd-client 3.8 from Ubuntu precise 12.04 old LTS [use old versions only locally for security!] – user1742529 Dec 25 '19 at 12:32
  • Simply specifying xz in place of /usr/lib/.../nbdkit-xz-plugin.so works great and lets it pick x86_64-linux-gnu or i386-linux-gnu automatically. – ManSamVampire Feb 06 '21 at 05:27
  • 1
    As of Ubuntu 20.04 (nbdkit v1.16.2), the nbdkit-xz-plugin.so file is missing. So, probably one should use --filter=xz as specified by @EkriirkE. Whereas, I simply downloaded the deb for nbdkit for Debian Buster, extracted the plugin from it and used it, and that worked too. – ManSamVampire Feb 06 '21 at 05:38