Migrate from software-RAID & EXT3 to RAID-Z on new hardware?

Question

I have an existing, live, Ubuntu server... supporting many services, that is working fine - but is installed on ageing hardware. I'd like to retain an (almost) identical configuration - on new hardware - but to adopt ZFS and RAID-Z in place of the existing software RAID and EXT-3 solution on the old server.

I've installed Ubuntu on the new server, and exposed it to my LAN (on a distinct static-IP) and established a suitable ZFS configuration. Migrating the configuration of numerous services (e.g. mail; web-apps; VPNs; proxies; firewalls; databases - etc. etc.) is proving more convoluted than I initially anticipated. With a 'manual' strategy to migrate, I envisage a much larger down-time than would be acceptable.

Are there any tools that would make this easier (and less error-prone)? Is there an easier strategy by which I could achieve the same end result?

Have you considered using rsync to duplicate everything from the current working server to the new one? — Elder Geek, Aug 27 '16 at 17:29
Is the software staying the same between the old and new servers? If you're staying with the same version of Ubuntu and the same versions of all packages, something like rsync may work. If you are also upgrading your software stack, it would be risky to just copy everything over wholesale and expect it all to work. — Charley, Aug 29 '16 at 05:40
Most of the software is going to stay the same - but not quite all. For example, I'm dumping everything to do with Raid-1 and adopting tools for ZFS. The servers are going to be 'very similar' - but I don't expect them to be identical. — aSteve, Aug 30 '16 at 17:51

score 2 · Answer 1 · edited Apr 13 '17 at 12:24

There are various ways to clone a machine. Make an image, and restore the image on other hardware. However, if you want to move your server to new hardware, with very little downtime (within some minutes), you can use the description in this article. The article assumes a Debian Linux system, but this will work with Ubuntu as well.

Install a base system

You will need a couple of things before you can start hotcloning:

a running minimal install (should take you max. 10 minutes) a running network card disks and partitions mounted the way you want them ssh and rsync installed No other running services (except for sshd, if you do this remote like in the example). stop inetd, atd, crond, klogd, syslogd

Synchronize the data

Using rsync we can synchronize old_machine to new_machine. Rsync only transfers differences, so you can stop and restart rsync any time, and you can also resync several days later without starting all over again, or rerun rsync to transfer all changed files.

Exclude patterns for rsync

Some files should not be transferred because they are hardware specific, or are available by default, or auto-generated.

These are specific for your hardware and your partition setup:

/boot/
/lib/modules
/etc/modules
/etc/lilo.conf
/etc/fstab
/etc/mtab

These are already available. /proc is a virtual filesystem provided by the running kernel, /dev is the device tree from the base install, lost+found are directories created during partition formatting:

/proc
/dev
lost+found/

These are excluded because you want to keep the logs separate for each machine, and you want (for now) to have a different network setup for each machine:

/var/log
/etc/network/interfaces

An exclude file contains all these entries, every entry on a new line. If this isn't clear see man rsync

Here's an example hotclone_exclude file which you'll store in the /root directory and reference with the rsync command:

/boot/
/lib/modules
/etc/modules
/etc/lilo.conf
/etc/fstab
/etc/mtab
/proc
/dev
lost+found/
/var/log
/etc/network/interfaces
/root

Warning: if you have network mounts you need to add them to the exclude file too!!!

Sync data

rsync -e ssh -avz --exclude-from=/root/hotclone_exclude old_server:/ /

Switch servers!

stop services on old_server

rerun rsync (should not take long since most transfers are done already)

update /etc/network/interfaces on new_server to contain the IP address of the old_server run ifdown -a on old_server run ifdown -a on new_server run ifup -a on new_server

Put the old server aside unchanged as a spare (better safe than sorry).

You should be all set.

Further note: If you need to exclude the files from a specific package you can obtain them with apt-file as in apt-file packagename which will give you a list of files in that package and their installation locations.

Example: $apt-file list trash-cli

trash-cli: /usr/bin/restore-trash
trash-cli: /usr/bin/trash
trash-cli: /usr/bin/trash-empty
trash-cli: /usr/bin/trash-list
trash-cli: /usr/bin/trash-put
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/__init__.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/assert_equals_with_unidiff.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/describe_trash_list.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/files.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/output_collector.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_file_descriptions.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_filesystem.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_persist.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_restore_trash.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_trash_empty.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/test_trash_put.py
trash-cli: /usr/lib/python2.7/dist-packages/integration_tests/trashinfo.py
trash-cli: /usr/lib/python2.7/dist-packages/trash_cli-0.12.7.egg-info/PKG-INFO
trash-cli: /usr/lib/python2.7/dist-packages/trash_cli-0.12.7.egg-info/SOURCES.txt
trash-cli: /usr/lib/python2.7/dist-packages/trash_cli-0.12.7.egg-info/dependency_links.txt
trash-cli: /usr/lib/python2.7/dist-packages/trash_cli-0.12.7.egg-info/entry_points.txt
trash-cli: /usr/lib/python2.7/dist-packages/trash_cli-0.12.7.egg-info/top_level.txt
trash-cli: /usr/lib/python2.7/dist-packages/trashcli/__init__.py
trash-cli: /usr/lib/python2.7/dist-packages/trashcli/cmds.py
trash-cli: /usr/lib/python2.7/dist-packages/trashcli/list_mount_points.py
trash-cli: /usr/lib/python2.7/dist-packages/trashcli/trash.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_available_trash_dirs.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_characterization.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_joining_paths.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_list_mount_points.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_parser.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_parsing_trashinfo_contents.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_trash.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_trash_new_tests.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_trash_put.py
trash-cli: /usr/lib/python2.7/dist-packages/unit_tests/test_trashdir.py
trash-cli: /usr/share/doc/trash-cli/changelog.Debian.gz
trash-cli: /usr/share/doc/trash-cli/copyright
trash-cli: /usr/share/man/man1/restore-trash.1.gz
trash-cli: /usr/share/man/man1/trash-empty.1.gz
trash-cli: /usr/share/man/man1/trash-list.1.gz
trash-cli: /usr/share/man/man1/trash-put.1.gz
trash-cli: /usr/share/man/man1/trash.1.gz
trash-cli: /usr/share/pyshared/integration_tests/__init__.py
trash-cli: /usr/share/pyshared/integration_tests/assert_equals_with_unidiff.py
trash-cli: /usr/share/pyshared/integration_tests/describe_trash_list.py
trash-cli: /usr/share/pyshared/integration_tests/files.py
trash-cli: /usr/share/pyshared/integration_tests/output_collector.py
trash-cli: /usr/share/pyshared/integration_tests/test_file_descriptions.py
trash-cli: /usr/share/pyshared/integration_tests/test_filesystem.py
trash-cli: /usr/share/pyshared/integration_tests/test_persist.py
trash-cli: /usr/share/pyshared/integration_tests/test_restore_trash.py
trash-cli: /usr/share/pyshared/integration_tests/test_trash_empty.py
trash-cli: /usr/share/pyshared/integration_tests/test_trash_put.py
trash-cli: /usr/share/pyshared/integration_tests/trashinfo.py
trash-cli: /usr/share/pyshared/trash_cli-0.12.7.egg-info/PKG-INFO
trash-cli: /usr/share/pyshared/trash_cli-0.12.7.egg-info/SOURCES.txt
trash-cli: /usr/share/pyshared/trash_cli-0.12.7.egg-info/dependency_links.txt
trash-cli: /usr/share/pyshared/trash_cli-0.12.7.egg-info/entry_points.txt
trash-cli: /usr/share/pyshared/trash_cli-0.12.7.egg-info/top_level.txt
trash-cli: /usr/share/pyshared/trashcli/__init__.py
trash-cli: /usr/share/pyshared/trashcli/cmds.py
trash-cli: /usr/share/pyshared/trashcli/list_mount_points.py
trash-cli: /usr/share/pyshared/trashcli/trash.py
trash-cli: /usr/share/pyshared/unit_tests/test_available_trash_dirs.py
trash-cli: /usr/share/pyshared/unit_tests/test_characterization.py
trash-cli: /usr/share/pyshared/unit_tests/test_joining_paths.py
trash-cli: /usr/share/pyshared/unit_tests/test_list_mount_points.py
trash-cli: /usr/share/pyshared/unit_tests/test_parser.py
trash-cli: /usr/share/pyshared/unit_tests/test_parsing_trashinfo_contents.py
trash-cli: /usr/share/pyshared/unit_tests/test_trash.py
trash-cli: /usr/share/pyshared/unit_tests/test_trash_new_tests.py
trash-cli: /usr/share/pyshared/unit_tests/test_trash_put.py
trash-cli: /usr/share/pyshared/unit_tests/test_trashdir.py

If you don't have it you can install it with sudo apt install apt-file

For some more useful tools see @drysdam answer here and @Ressu answer here

Sources:

http://olivier.sessink.nl/publications/hotcloning/

http://olivier.sessink.nl/publications/hotcloning/exclude.txt

man rsync

https://superuser.com/questions/82923/how-to-list-files-of-a-debian-package-without-install

That's all good advice on using rsync... I'm apprehensive because the fresh install isn't quite the same as the install from which I'm migrating. I don't want to migrate anything relating to periodic verification of /dev/md0 (the Raid-1 mirror) for example. I want to prefer those packages in the new base install over those in the old. — aSteve, Aug 30 '16 at 17:49
you'll note that `/dev/ is excluded. you can also exclude any related packages you want in a similar fashion. — Elder Geek, Aug 30 '16 at 17:59
/dev/* would need to be excluded as that folder just contains devices. I'm not clear how to exclude (Ubuntu) packages - as a typical package does not reside at a single path in the file-system. — aSteve, Aug 30 '16 at 19:20

Migrate from software-RAID & EXT3 to RAID-Z on new hardware?

1 Answers1