Cloning a Ubuntu installation (including repositories, held packages & keys)

Question

From How to list all installed packages we learn how to backup list of installed packages.

From How to See list of Repository? we learn how to backup list of installed repositories.

But how do we automatically add all repositories before we run a script to install all installed packages from the machine we are cloning?

Then how do we take the list of all installed packages and parse it to install them?

Before that how do we know what keys we need to get?

Also how do we generate list of held packages to backup and then reapply to cloned machine?

I've heard of apt-clone but not sure if it does all the above.

My list of repositories are:

$ cd /etc/apt/sources.list.d/ && cat *.list
# deb http://ppa.launchpad.net/colingille/freshlight/ubuntu xenial main # disabled on upgrade to xenial
# deb-src http://ppa.launchpad.net/colingille/freshlight/ubuntu trusty main
deb http://ppa.launchpad.net/fossfreedom/indicator-sysmonitor/ubuntu xenial main
# deb-src http://ppa.launchpad.net/fossfreedom/indicator-sysmonitor/ubuntu xenial main
deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main
### THIS FILE IS AUTOMATICALLY CONFIGURED ###
# You may comment out this entry, but any other modifications may be lost.
deb http://dl.google.com/linux/earth/deb/ stable main
deb http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial main
# deb-src http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial main
# deb-src http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial main
deb https://download.01.org/gfx/ubuntu/16.04/main xenial main #Intel Graphics drivers
deb http://ppa.launchpad.net/mkusb/ppa/ubuntu xenial main
# deb-src http://ppa.launchpad.net/mkusb/ppa/ubuntu xenial main
deb http://ppa.launchpad.net/peek-developers/stable/ubuntu xenial main
# deb-src http://ppa.launchpad.net/peek-developers/stable/ubuntu xenial main
deb http://ppa.launchpad.net/peterlevi/ppa/ubuntu xenial main
# deb-src http://ppa.launchpad.net/peterlevi/ppa/ubuntu xenial main
deb http://ppa.launchpad.net/sethj/silentcast/ubuntu xenial main
# deb-src http://ppa.launchpad.net/sethj/silentcast/ubuntu xenial main

My list of packages is 100k and won't fit into this message but, here is a subset:

$ cat ~/.packages
a11y-profile-manager-indicator          install
account-plugin-aim              install
account-plugin-facebook             install
account-plugin-flickr               install
account-plugin-google               install
account-plugin-jabber               install
account-plugin-salut                install
account-plugin-twitter              install
account-plugin-windows-live         deinstall
account-plugin-yahoo                install
accountsservice                 install
acl                     install
acpid                       install
adduser                     install
adium-theme-ubuntu              install
adobe-flash-properties-gtk          install
adobe-flashplugin               install
adwaita-icon-theme              install

About 5% of the list have been removed and show deisntall instead of install.

---

EDIT 1 based on comment instructions these are the keys which are contained in /etc/apt/trusted.gpg and /etc/apt/trusted.gpg.d which I've added to my daily backup:

$ sudo apt-key list
/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>
sub   2048g/79164387 2004-09-12
pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>
pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>
pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>
pub   1024D/7FAC5991 2007-03-08
uid                  Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub   2048g/C07CB649 2007-03-08
pub   4096R/D38B4796 2016-04-12
uid                  Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   4096R/640DB551 2016-04-12 [expires: 2019-04-12]
sub   4096R/997C215E 2017-01-24 [expires: 2020-01-24]
pub   2048R/63961D39 2016-01-25
uid                  Elio Martinez (Installer compilation) <elio.martinez.monroy@intel.com>
sub   2048R/98E81FA7 2016-01-25
/etc/apt/trusted.gpg.d/colingille-freshlight.gpg
pub   1024R/CB7F5C71 2010-12-30
uid                  Launchpad PPA for Congelli501
/etc/apt/trusted.gpg.d/fossfreedom_ubuntu_indicator-sysmonitor.gpg
pub   1024R/F4FE239D 2012-01-05
uid                  Launchpad PPA for fossfreedom
/etc/apt/trusted.gpg.d/graphics-drivers_ubuntu_ppa.gpg
pub   4096R/1118213C 2015-08-12
uid                  Launchpad PPA for Graphics Drivers Team
/etc/apt/trusted.gpg.d/mkusb_ubuntu_ppa.gpg
pub   4096R/54B8C8AC 2014-08-14
uid                  Launchpad PPA for MKUSB
/etc/apt/trusted.gpg.d/peek-developers_ubuntu_stable.gpg
pub   4096R/76BAFBC6 2017-02-14
uid                  Launchpad PPA for Peek Developers
/etc/apt/trusted.gpg.d/peterlevi_ubuntu_ppa.gpg
pub   1024R/A546BE4F 2012-06-28
uid                  Launchpad PPA for Peter Levi
/etc/apt/trusted.gpg.d/sethj_ubuntu_silentcast.gpg
pub   4096R/A1A7B3E6 2014-10-03
uid                  Launchpad PPA for Seth Johnson

---

EDIT 2

As per comments I gave aptitude-build-state-bundle and aptitude-run-state-bundle a try after installing programs using:

sudo apt install aptitude-common

Running the build program resulted in a whopping 334 MB file compressed to 92 MB.

Transferring the compressed file to the target laptop and running aptitude-run-state-bundle gave this colorful screen (among many):

We can see it's tracking held packages! The old laptop was running kernel 4.4 and the new laptop defaulted to 4.10. I deviated from the unwritten rule of copying and pasting text screens and used color snapshot because it's easier to digest.

This is about as far as I got because an on-line tutorial for aptitude-run-state-bundle hasn't been located yet.

I did discover that my daily backup wasn't including permissions and had to be restored in two passes, once with sudo, once without. I discovered I wasn't backing up custom Grub fonts and Plymouth splash screens. I discovered I wasn't backing up Nautilus custom scrips either.

A full system backup and restore is tempting. It will avoid all these piecemeal steps. However on the old laptop grub is setup with gparted and CSM, whilst on the new laptop grub is using GPT and UEFI. I'm not sure what headaches that might cause.

For sure /etc/fstab must NOT be restored. Besides different UUIDs, the source has a regular SSD (/dev/sdc3) and the target has an NVMe SSD (/dev/nmve0p5).

---

EDIT 2 November 1, 2017

Selecting packages from source machine and installing them on target machine didn't bring over conky which was held back to version 1.9. Other software like unity-tweak-tool didn't reinstall either.

Conky script required extensive rework for temperature. Intel i7 6700HQ has different temperature thermal zones than i7 3630QM. vnStat was different because eth0 is no longer used as the new laptop has a different adapter name.

Old laptop has Super VGA, HDMI and built in LVDS that can be either iGPU (HD4000) or discrete graphics (nVidia GE Force GT650M) controlled. It was a shocker to learn on new laptop HDMI is hardwired to nVidia GTX 970M and the iGPU (HD530) can only control LVDS and ThunderBolt3 to DisplayPort to HDMI adapter.

Further complications that WSL (Windows Subsystem for Linux) cannot write to Linux partitions and Linux can only write to Windows NTFS partitions but doesn't support not WSL hybrid directories. Windows apps cannot write to WSL hybrid directories either. WinBTRFS is not released yet and Ext2FSD version .69 bricks Windows 64 bit drives. WSL only works with Windows 64-bit.

Simple save and restore is off the table.

---

Conclusion

December 10, 2017

• At the end of the day all applications were installed manually.
• HDMI had no audio so setpci had to be used during systemd boot.
• Laptop wouldn't suspend due to NVMe SSD so ACPI hotplugging had to be disabled.
• nVidia screen tearing was solved with NVIDIA X Server Settings composition pipeline.
• A new mount (/mnt/e) on SSD WSL and Ubuntu 16.04.