4

I have a very large server (Minecraft) using tens of gigabytes of files. Every two days, I make a backup, but since the server is so big, I have to shut it down for 45 minutes until the backup is done.

If I backup while the server is running, the backup folder gets corrupted because not all files were backed up at the same time and there's duplicate data, missing data and other problems. Chmodding the server folder recursively to 440 (read-only) makes the server spam thousands of errors, and sometimes crashes it or corrupts data.

What I am trying to do is to switch the server to read-only mode, but to allow it to write to temporary "journal" files which will be written to the real files once the backup is finished.

Is this possible at all? If yes, what tool should I use to do this? I was thinking something such switching the server folder to a soft link to a folder or partition that simulates this.

2 Answers2

1

If you created your filesystem with Logical Volume Manager (LVM) enabled, then you could make a filesystem snapshot and then backup that snapshot while your server is never taken down.

Three-page tutorial here: http://www.howtoforge.com/linux_lvm_snapshots

Although it could mean that you will have to recreate your filesystem.

Majal
  • 7,711
1

Using overlayfs

Use overlayfs on the root of your server (server folder).

  1. Make a read-only link to the server folder, from which you can back up the data while the real directory is covered

    mkdir -p /path/to/ReadOnlyServerRoot sudo mount --bind /path/to/serverRoot /path/to/ReadOnlyServerRoot -o remount,ro

  2. Shutdown the server so that files and databases are properly closed

  3. Protect actual server root, /path/to/serverRoot, from write.

    sudo mount -t overlayfs overlayfs /path/to/serverRoot -o rw,uppderdir=/path/to/changes,lowerdir=/path/to/serverRoot

The changed and newly added data is stored at /path/to/changes. When a file is modified a new copy is made on /path/to/changes/path/to/file and this file is updated.

  1. Start the server

  2. Now you can backup your data from /path/to/ReadOnlyServerRoot

  3. Shutdown server after the backup is completed

  4. Remove the cover

    sudo umount /path/to/serverRoot

  5. Use a synchronization software, such as rsync, to merge the change data into the server file.

    rsync /path/to/changes /path/to/serverRoot
rm -rf /path/to/changes

  6. Start the server

More information on on overlayfs's option can be found on this answer.

Steps 2, 4, 6, and 9 are optional, but highly recommended: At the time of creating and removing overlay on server folder it is highly recommended to shutdown your server (only seconds of downtime) for a successful backup as server may cache partial data on their memory and only a part of the data would be updated. As a result backup contains invalid/unusable data which in turn lead to lost data.

Using a file system snapshot

switch to a file system that support snapshot like btrfs or lvm

Update

If overlayfs is not available use aufs or unionfs
unionfs 

    mount -t unionfs -o dirs=/branch_rw=rw:/branch_ro=rounionfs/union

creates a Union in directory /union with the branch directories /branch_rw(writable) and /branch_ro (read-only).

aufs 

mkdir /tmp/dir1
mkdir /tmp/aufs-root
mount -t aufs -o br=/tmp/dir1:/tmp/dir2 none /tmp/aufs-root/

The first two commands created 2 new directories. The mount.aufs is the command to mount the filesystem as Union mount. The mount command, specifies it is going to union mount /tmp/dir1 and /tmp/dir2 under /tmp/aufs-root. The directory /tmp/aufs-root will have the content of both /tmp/dir1 and /tmp/dir2.

Community
  • 1
totti
  • 6,818
  • 4
  • 39
  • 48
  • Won't it create a loop if the lowerdir is the same as the mountpoint? –  Jun 23 '14 at 17:29
  • No. The lower dir is hidden under the current mount. There is a error in the answer: actually the readonly server folder is not available and hence you can't backup. Easy solution is to rename server folder before overlaying and create a new folder as server folder then mount there. I'm trying to find a better solution after that I will update the answer. – totti Jun 23 '14 at 18:16
  • @Runemoro. Fixed and updated – totti Jun 24 '14 at 08:06
  • @Runemoro, How is it ? – totti Jun 25 '14 at 14:12
  • It works, but it doesn't really mount the cover in read only, I get this message: "mount: warning: cover seems to be mounted read-write. ". –  Jun 26 '14 at 10:09
  • @Runemoro, at which step ? – totti Jun 26 '14 at 10:18
  • When I create the bind. –  Jun 26 '14 at 10:24
  • @Runemoro, at step 3. Is it ? Then no problem because the cover is mounted RW otherwise most server will crash. Did your server run in read only mode? – totti Jun 26 '14 at 10:24
  • Also, when I test it on the server (not on my computer), I get an error telling me that mount: special device overlayfs does not exist, but on my laptop it had worked. They both have ubuntu trusty with the server tasksel and ubuntu-standard package. –  Jun 26 '14 at 10:26
  • @No, when I create the read-only server root. –  Jun 26 '14 at 10:28
  • RW mode on at bind will not be a problem as the server don't accessing from /serverRoot /path/to/ReadOnlyServerRoot. It even know about it. – totti Jun 26 '14 at 10:33