2

After spending time trying to make rsync work, I decided to use something else. I have a tree of files at /home that I want to backup to a 64G USB stick once a day, using the same tree structure, but only copy over the ones that don't exist in destination ( and preserve timestamps ).

The problem:

Even if a source folder has only one new file (and 400 old ones), my rsync always finds a reason to copy about half of them over to dest.

I experimented with something along the lines of this

find /home/me/a/r/z/ -type f -mmin 400 | xargs -0 -I{} cp '{}' /media/usb-id/$( cut -d / -f 4- {} )

Lots of mistakes, I know. I use "4" with cut because that's where the tree starts in destination (/media/usb-id/r/z/whatever). I think I should be using sed somehow in all of this. Any ideas?

Obvious error:

cut: {}: No such file or directory
pLumo
  • 26,947
YetAnotherNiebie
  • 23
  • 2
    Could you tell us what do you want to have? Backup one folder to external drive? What kind of file system do you have on this external drive - ntfs? – mariaczi May 25 '18 at 13:01
  • 2
    I'm pretty sure rsync works correctly and you should rather ask how to make it work correctly for you instead of asking for an alternative. Rsync is skipping files based on modification time & file size. Maybe have a look on your usb if these somehow changed and why. As an alternative try to use -checksum switch. This will be slower but in case your mod time on your stick changes whyever this might help. – pLumo May 25 '18 at 13:10
  • How do you know which files are actually copied over by rsync? If you mean the output of rsync -P, that’s supposed to show every file IIRC – it has to loop over every file to check for changes, obviously. – dessert May 25 '18 at 13:17
  • When using rsync for backups, you might find the options -RavC to be a good formula. – glenn jackman May 25 '18 at 13:19
  • Its about updating a tree structure in a usb stick ( vfat ) starting at /r ( /r/z, /r/t, /r/z/x etc ) using a similar structure from /dev/sda1 ( Ubuntu 10.04 ) as a source. I know rsync is the way to go but now that I've gone down this road it would be interesting to see how this can be implemented in an other, not so intuitive way. – YetAnotherNiebie May 25 '18 at 13:20
  • 2
    Probably the timestamp resolution is not the same on the source and destination filesystems. rsync has an option for that case (--modify-window=) – Jean-Marie May 25 '18 at 13:24
  • 2
    Are you going from ext4 to ext4? Using Windows formats like NTFS or FAT32 is not recommended as it does not support ownership & permissions. Not sure then about timestamps. – oldfred May 25 '18 at 13:30
  • ext4 to vfat. Tried formatting the thing to ntfs, copying the files over the first time was way too slow. – YetAnotherNiebie May 25 '18 at 13:36
  • As oldfred states change your USB from FAT to ext4 and only changed/new files will be backed up. – WinEunuuchs2Unix May 25 '18 at 15:12
  • 1
    Likely related: rsync not working between NTFS/FAT and EXT – David Foerster May 25 '18 at 15:55

2 Answers2

0

sudo rsync -Havn between ext4 file systems

is the best way to go, if you want to preserve permissions and ownership of the files. This can be very important, because the linux system depends on it. But it is not important for personal data files.

I would recommend to start with a 'dry run'

sudo rsync -Havn source/ target

and if it looks good, remove the option n and do it,

sudo rsync -Hav source/ target

It preserves 'everything', so you can copy a whole linux operating system from one ext4 file system to another ext4 file system. And you can copy only a limited directory tree that way too.

An alternative to ext4 on the target drive is udf, if you must be able to read it from Windows. See this link. The udf file system can manage the ownership and permissions (and time stamps) of linux file systems.

But there is a drawback with udf. There is no well known tool to repair it.

sudodus
  • 46,324
  • 5
  • 88
  • 152
  • 1
    Thanks everyone for their replys, all very helpfull. I still find it interesting that everybody is such a huge fan of rsync and not really interested in experimenting with an alternative way to fix this, like a find/xargs combination. I thought it might be fun. I will try the -checksum and -modify-window options and probably end up formatting the stick to ext4, because for some reason ext4 to ntfs file transfers started out at 7MB/sec and ended up at 90kb/sec... – YetAnotherNiebie May 25 '18 at 16:17
  • It is also my experience, that linux writes slowly to NTFS. NTFS is a proprietary file system of Microsoft, and the linux driver for it is probably some kind of reverse engineered code, that it not as efficient as the free drivers for linux file systems. (FAT32 is also owned by Microsoft, but the original FAT code was probably free (long ago) and also rather simple, so the linux drivers for FAT can be quite efficient.) - Anyway, many of us like rsync, and we hope that you will not only get along with it, but start liking it too :-) – sudodus May 25 '18 at 19:21
  • Don't get me wrong, I like rsync just fine. I just want to get the job done, when I want it done, no matter what the tool is. Didnt have the time or energy to keep on typing commands that didnt work. Plus the vastness of Linux still overwhelms me occasionally. – YetAnotherNiebie May 25 '18 at 20:19
  • You are welcome @YetAnotherNewbieSort-of :-) – sudodus May 25 '18 at 20:32
0

You might have a look at this: https://github.com/Fitus/Zaloha.sh. It is a find-based shell script, also based on the same idea as yours (but it is a little bit more complex than your one-liner, as you might expect) ...

Petas
  • 1