3

TLDR

How do I create a bootable Ubuntu Live USB that has a compressed filesystem.squashfs of size greater than 4GB, specifically 4,294,967,295 bytes, the limit of a file for drives formatted in FAT32.

More Detail

I'm trying to create a customized bootable Ubuntu Live USB that does not persist data, but am running into limitations:

What I've Tried

  • Cubic which is fantastic, but once I'm done customizing the image the filesystem.squashfs can be > 4GB in size, even when using XZ compression
  • Using this script to build the filesystem.squashfs from a currently running installation (preferred) using mksquashfs, but I run into the same filesize limit for FAT32 even when using XZ compression

Desired Outcome

  • I want a Live Ubuntu USB that does not persist between sessions
  • I want to be done with this filesize limit without having to worry about trimming packages or removing installed packaged (I've already ran things like apt clean/autoclean etc)
  • I want to customize and add as many packages as I want, either through some application like Cubic or from some live-running Ubuntu installation without worrying about the squashed FS filesize

To Summarize

  • Is it possible to split the filesystem.squashfs into multiple files that casper's menuentry linux reference to the squashed FS will understand?
  • Are there alternatives to read-only filesystems other than mksquashfs that are compatible with grub/casper?
  • Is this possibly a chance to write my own package to solve this problem, or would this be too much of an undertaking?

2 Answers2

2

I was able to figure this out. UEFI cannot read from anything except FAT, but casper can. Here's instructions for how to create a Bootable Live USB with a squash FS > 4GB. This method does NOT destroy the whole USB unlike other Bootable USB creation methods.

Environment

  • OS: Ubuntu 22.04
  • ISO Creator: Cubic
  • Source Image: Latest 22.04 Desktop downloaded from ubuntu.com

Bootable USB Partitioning

  • Partition 1
    • Format: FAT32
    • Size: ~5GB
    • Flags: boot
  • Partition 2
    • Format: ext4
    • Size: 10-15GB (enough size for the squashfs filesystem)

The rest of the USB can be partitioned as you like without removing any data during installation. Be sure to take note of the device mounts for your partitions, for example /dev/sda1 (FAT32), /dev/sda2 (ext4).

Instructions

  1. Create your customized ISO using Cubic.

  2. Follow instructions until you get to the Options page.

This is where we tell casper to load the squashed FS from a different location than the FAT32 boot partition.

First, we want to reference the UUID of the ext4 partition instead of it's device ID (/dev/sda2 in my case) since the device ID can change.

You can find the UUID of the ext4 partition you created using these instructions: https://linuxhint.com/uuid_storage_devices_linux/. Since my ext4 partition is /dev/sda2, I can do this to get my ext4 UUID:

$ ll /dev/disk/by-uuid | grep sda2
lrwxrwxrwx 1 root root  10 Oct 10 20:38 727cac18-044b-4504-87f1-a5aefa774bdb -> ../../sda2

and this shows that my ext4 partition's UUID is 727cac18-044b-4504-87f1-a5aefa774bdb.

Modify the grub config in the Boot tab. You'll need to modify the first menuentry like so:

menuentry "Ubuntu RAM" {
    set gfxpayload=keep
    linux   /casper/vmlinuz boot=casper live-media=/dev/disk/by-uuid/727cac18-044b-4504-87f1-a5aefa774bdb toram file=/cdrom/preseed/ubuntu.seed quiet splash noprompt --- 
    initrd  /casper/initrd.gz
}

Be sure to change /dev/disk/by-uuid/<UUID> to the correct ext4 partition UUID you found during this step!

Casper boot option documentation can be found here.

We've added the following boot options

  • toram (note: remove this boot option if you do not have the RAM to hold the entire squashfs filesystem)
  • noprompt
  • live-media=<DEVICE_UUID_PATH>

We've removed the following boot options

  • maybe-ubiquity
  1. When prompted for compression method on the Compression Page, feel free to choose LZ4 as this is fastest and filesystem.squashfs file size no longer matters.

  2. Generate your custom ISO.

  3. Extract your custom ISO to some empty directory like ~/Documents/my-ubuntu-folder.

  4. Copy all files from the extracted ISO except for casper/filesystem* to the FAT32 partition. casper/vmlinuz and casper/initrd.gz must be on the FAT32 boot partition.

You might see some warning of inability to copy symlinks because FAT32 doesn't support them; this is OK and you can skip these files while maintaining usability of the bootable USB.

  1. Create a directory named casper in your ext4 partition and copy all extracted casper/filesystem* files to this directory.

Once finished, the casper directory on your FAT32 partition should look like this:

$ ls -1 casper/
initrd.lz  
vmlinuz

And the casper directory on your ext4 partition should look like this

$ ls -1 casper/
filesystem.manifest
filesystem.manifest-minimal-remove
filesystem.manifest-remove
filesystem.size
filesystem.squashfs

You've now created a UEFI bootable USB that uses a separate storage partition for its read-only filesystem via casper's live-media boot option.

  • Glad you figured that out. I simply splitted file system when rebuilding iso file. I did it after reading casper scripts and seeing IIRC it works (mounts via overlayfs as root) with all files in casper folder with squashfs extension. – Alex Martian Feb 18 '23 at 10:48
  • Initially I was confused you talk about FAT as afaik ISO file is some CD/DVD format ISO9660. After reading carefully I see you extract ISO to FAT. Well at least afaik you are incorrect stating UEFI only boot fat, no it can boot ISO9660 too! However with ISO9660 I seem to hit that limit too around 4.2Gb which you mentioned. BTW I've only recently done that (squashfs split) and now testing stability of such system. – Alex Martian Feb 18 '23 at 10:59
  • More detail on why I use FAT instead of ISO, from my experience ISO writes over the whole drive, by extracting the files to a FAT32 partition I can do what I want with the rest of the drive without destroying it. – Bilbo Baggins Dec 26 '23 at 16:09
2

AFAIK as far as I've seen casper scripts from initramfs file they make new root via overlayfs from all files ending with .squashfs located in casper folder on ISO.

So I simply splitted file system when rebuilding ISO file and it works! I've only recently done that (squashfs split) and now testing stability of such system.

The way OP solved the issue is interesting too, split's benefit is that with ISO file one can use tools such as Ventoy to have multiple systems on one USB stick.

P.S. Disclaimer: try and see for your ISO file, if does not work out maybe on ISO you try to customize casper scripts are different.

  • I finally got around to attempting this using the split command but got errors during boot. Could you possibly provide more detail on this squashfs split command in your answer? – Bilbo Baggins Dec 26 '23 at 16:07
  • @BilboBaggins, here is code I've developed to make changes to contents of ISO file and repack it again, compress2squashfs function calls mksquashfs with parameters (called several times to 'split' the system), https://github.com/mars4science/Bash-scripts-to-enhance-amend-Linux-distro-before-and-after-installation/blob/main/_make_custom_liveusb.sh – Alex Martian Jan 01 '24 at 12:11