Custom Ubuntu image is broken after Grub update (when using UEFI)

Question

So I've been working on creating a custom Ubuntu 16.04.5 image. I follow the instructions here to chroot into the file system that gets put onto the computers and do apt update && apt full-upgrade. After that I package everything up into an ISO and use that to image a computer. I do most of my testing in VirtualBox & when I tell the VM to boot using UEFI, the Ubuntu installation gives me an error:

The 'grub-efi-amd64-signed' package failed to install into /target/.
Without the GRUB boot loader, the installed system will not boot.

So I went back to the chroot environment, installed the 'grub-efi-amd64-signed' package (there's no internet connection during the install process), and all is well. This was working fine for me until recently. Now, after the installation I'm now taken to the Grub command prompt.

Now it seems there's an update to Grub2; from 2.02~beta2-36ubuntu3.18 to 3.19. If I don't install that update I can't use UEFI because of that error I get. If I install the 'grub-efi-amd64-signed' then if forces the other Grub packages to update. If I DO install that package & all the updates, then the computer boots to the Grub command prompt. I really don't know where to begin to fix this. What can I do to update Grub & fix the issue?

Could this be because the ISO is using the previous version of Grub?

EDIT: Something else I've noticed. I'm able to get everything to install (and boot) correctly if I apt-mark hold on the following packages:

linux-generic-hwe-16.04
linux-image-generic-hwe-16.04
linux-headers-generic-hwe-16.04
linux-signed-generic-hwe-16.04
grub-common
grub2-common

After that, I can apt full-upgrade, create the ISO & do the install... So why does it break when I update the kernel & Grub?

FWIW, here's my preseed file:

### Localization
d-i debian-installer/locale                       string en_US
d-i console-setup/ask_detect                      boolean false
d-i console-setup/layoutcode                      string us


### Network Configuration
d-i netcfg/dhcp_timeout                           string 300
d-i netcfg/dhcp_failed                            note
d-i netcfg/dhcp_options                           select Do not configure the network at this time
d-i netcfg/get_hostname                           string dev
d-i netcfg/get_domain                             string example.com


### Account Setup
d-i passwd/user-fullname                          string User
d-i passwd/username                               string user
d-i passwd/user-password-crypted                  password $6$...
d-i user-setup/allow-password-weak                boolean true
d-i user-setup/encrypt-home                       boolean false


### Clock and Time Zone Setup
d-i clock-setup/utc                               boolean true
d-i time/zone                                     string US/Eastern
d-i clock-setup/ntp                               boolean true


### Partitioning and Encryption
# Use LVM for encryption
d-i partman-auto/method                           string crypto

# Suppress LVM and RAID warnings about previous configurations
d-i partman-lvm/device_remove_lvm                 boolean true
d-i partman-md/device_remove_md                   boolean true

# Confirm writing on existing partitions
d-i partman-lvm/confirm                           boolean true

# Use the entire logical volume
d-i partman-auto-lvm/guided_size                  string max

# Have all files on this partition only
d-i partman-auto/choose_recipe                    select atomic

# Specify ext4 since the default is ext3
d-i partman/default_filesystem                    string ext4

# Begin the partitioning without user interaction
d-i partman-partitioning/confirm_write_new_label  boolean true
d-i partman/choose_partition                      select finish
d-i partman/confirm                               boolean true
d-i partman/confirm_nooverwrite                   boolean true

# Use the encryption key to perform the encryption
partman-crypto partman-crypto/passphrase          string password
partman-crypto partman-crypto/passphrase-again    string password

# Agree to use weak passphrase and confirm if prompted
partman-crypto partman-crypto/weak_passphrase     boolean true
partman-crypto partman-crypto/confirm             boolean true


### Package Selection
tasksel tasksel/first                             multiselect standard
tasksel tasksel/first                             multiselect ubuntu-desktop
d-i pkgsel/updatedb                               boolean false


### Boot Loader Installation
d-i grub-installer/only_debian                    boolean true


### Finishing Up the Installation
ubiquity ubiquity/reboot                           boolean true

Answer 1

I believe you are running into the same issue found here: 16.04 new installation gives grub-efi-amd64-signed failed installation /target/ ubuntu 16.04 at the end

I think you need to create the /boot/efi partition manually and mark it with a boot flag. We ran into this recently and resolved with preseeds like below, edit for your drive if needed. This will create the correct partitions, use the entire disk (up to 1000 TB) and swap (8 GB or 2xRAM):

d-i partman-auto/disk                            string /dev/sda
d-i partman-auto/method                          string crypto
d-i partman-lvm/device_remove_lvm                boolean true
d-i partman-lvm/device_remove_lvm_span           boolean true
d-i partman-auto/purge_lvm_from_device           boolean true
d-i partman-auto-lvm/new_vg_name                 string system
d-i partman-auto/expert_recipe string         \
   boot-crypto ::                             \
      1 1 1 free                              \
         $bios_boot{ }                        \
         method{ biosgrub } .                 \
      256 256 256 fat32                       \
         $primary{ } $lvmignore{ }            \
         method{ efi } format{ } .            \
      512 512 512 ext3                        \
         $primary{ } $bootable{ }             \
         method{ format } format{ }           \
         use_filesystem{ } filesystem{ ext3 } \
         mountpoint{ /boot } .                \
      2000 10000 1000000000 ext4              \
         $lvmok{ }                            \
         method{ format } format{ }           \
         use_filesystem{ } filesystem{ ext4 } \
         mountpoint{ / } .                    \
      8000 8000 200% linux-swap               \
         $lvmok{ }                            \
         method{ swap } format{ } .
d-i partman-lvm/confirm                          boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition                     select finish
d-i partman/confirm                              boolean true
d-i partman/confirm_nooverwrite                  boolean true