Improve full backup speed with Déjà Dup / Duplicity

Question

Is there an ease fix two to improve the backup speed using the default Déjà Dup Backup solution of the Ubuntu Gnome Desktop?

Actually I'm quite happy with my weekly duplicity backups, but I set it up that I have every three month a full backup to my local NAS. This takes around two full days for ~1TB of data.

I believe there is at least two bottlenecks of my duplicity solution:

1. the backup runs only one core and
2. it uploads just small chunks of 50 MB to the NAS.

Especially the second issue means, that I can't use the full capacity of my Gigabit-LAN. Duplicity uploads the files with about 25 MiB/s. I upload manually a bigger file I get easily 80 MiB/s (NAS is connected with a 2 x 1 GbE bond).

Here is a typical screenshot of the System Monitor:

Does anybody have an idea, how to improve the full backup speed? Eg. can I change the chunk size etc.

Answer 1

Found it, Deja Dup overwrites the default duplicity chunk size. The source code explains the advantages and disadvantages:

  // Returns volume size in megs
  int get_volsize()
  {
    // Advantages of a smaller value:
    // * takes less temp space
    // * retries of a volume take less time
    // * quicker restore of a particular file (less excess baggage to download)
    // * we get feedback more frequently (duplicity only gives us a progress
    //   report at the end of a volume) -- fixed by reporting when we're uploading
    // Downsides:
    // * less throughput:
    //   * some protocols have large per-file overhead (like sftp)
    //   * the network doesn't have time to ramp up to max tcp transfer speed per
    //     file
    // * lots of files looks ugly to users
    //
    // duplicity's default is 25 (used to be 5).
    //
    // For local filesystems, we'll choose large volsize.
    // For remote FSs, we'll go smaller.
    if (DejaDup.in_testing_mode())
      return 1;
    else if (backend.is_native())
      return 50;
    else
      return 25;
  }

I managed to compile a custom version of deja-dup. Therefore, I had to do the following steps:

1. Clone source code

   git clone https://gitlab.gnome.org/World/deja-dup.git
   cd ./deja-dup
   

2. Switch to older branch with gtk3 support for Ubuntu 20.04

   git checkout --track origin/40
   

3. Patch the return of the get_volsize() funktion in DuplicityJob.vala and increase the version number. Eg.

   git apply patch40-7-1.diff
   

   patch40-7-1.diff (0.82 KB):

   diff --git a/libdeja/tools/duplicity/DuplicityJob.vala b/libdeja/tools/duplicity/DuplicityJob.vala
   index a229e8b0..bb6f77fe 100644
   --- a/libdeja/tools/duplicity/DuplicityJob.vala
   +++ b/libdeja/tools/duplicity/DuplicityJob.vala
   @@ -1399,9 +1399,9 @@ internal class DuplicityJob : DejaDup.ToolJob
        if (DejaDup.in_testing_mode())
          return 1;
        else if (backend.is_native())
   -      return 50;
   +      return 500;
        else
   -      return 25;
   +      return 250;
      }
   void disconnect_inst()
   diff --git a/meson.build b/meson.build
   index 266d7a36..fed8434d 100644
   --- a/meson.build
   +++ b/meson.build
   @@ -4,7 +4,7 @@
   SPDX-FileCopyrightText: Michael Terry
   project('deja-dup', ['vala', 'c'],
   
   version: '40.7',
   
   
   version: '40.7.1',
   license: 'GPL-3.0-or-later',
   meson_version: '>= 0.47'
   
   

4. Make and install (including installing missing packages):

   meson . _build
   ninja -C _build
   ninja -C _build install
   

Voilà

---

Update about performance

I just double-checked the speed of a full backup, it improved like 50% (from 24 to about 12 hours). Unfortunately, that is still not perfect as it packs each package and upload it before it starts the next one. But still better than before...