How can I view a progress bar when running rsync?

Question

I am using Ubuntu 12.04 as a repo and would like to view a progress bar when using rsync from the command line. I tried the option suggested in this article (-P), but I prefer to see a progress bar and not use Grsync. I am using rsync -P source dest currently.

I appreciate the fast response. I look forward to researching this. — Joseph Smith, Apr 14 '15 at 19:26
That was a good start but I need to know how to make the progress bar update as rsync is working. A friend said that I need to implement the server size and implement a watch command with pipe. — Joseph Smith, Apr 14 '15 at 19:55

Jon · Answer 1 · 2017-04-23T11:47:47.590

rsync has a --info option that can be used to not only output the current progress, but also the transfer rate and elapsed time:

--info=FLAGS            fine-grained informational verbosity

The explanation of how to use it comes under the -P option in the man page:

-P     The -P option is equivalent to --partial --progress.  Its purpose is to
       make it much easier to specify these two options for a long transfer that
       may be interrupted.

       There is also a --info=progress2 option that outputs statistics based on
       the whole transfer, rather than individual files.  Use this flag
       without  out‐putting  a  filename  (e.g. avoid -v or specify --info=name0)
       if you want to see how the transfer is doing without scrolling the screen 
       with  a  lot  of names.   (You  don’t  need  to specify the --progress
       option in order to use --info=progress2.)

So the following:

rsync -r --info=progress2 --info=name0 "$src" "$dst"

Results in the following being output and continuously updated:

18,757,542,664 100%   65.70MB/s    0:04:32 (xfr#1389, to-chk=0/1510)

Note that when the transfer starts the total number of chunks, and therefore the current progress, can change when the recursive option is used as more files are discovered for syncing

hmm my --info=progress2 progress bar shows ~8minutes remaining for the last 20h and jumps from 99 to 97, 98, 96 percent up and down? what is the purpose of this? :D — florian, Aug 28 '20 at 16:57
Quick note: the --no-i-r option disables incremental recursion, meaning that rsync will find all files before starting to transfer. This makes the percentage more accurate, but I would recommend only using it for local transfers. See https://serverfault.com/questions/219013/showing-total-progress-in-rsync-is-it-possible — Caleb Reister, Jul 02 '21 at 05:30
Based on the above comments, the following parameters have worked best for me:
rsync -az /source /destination --info=progress2 --no-i-r

Output:

1,063,226,358 20% 19.50MB/s 0:00:51 (xfr#91160, to-chk=189838/296682) — jorisw, Jul 15 '22 at 16:39

score 49 · Answer 2 · edited Sep 30 '22 at 19:26

49

You can use --progress and --stats parameters.

$ rsync -avzh --progress --stats root@server:/path/to/file output_name
root@server's password: 
receiving incremental file list
file
         98.19M  54%    8.99MB/s    0:00:08

From man page/explainshell:

-a, --archive
-v, --verbose
-z, --compress
-h, --human-readable
--progress  This  option  tells  rsync to print information showing
 the progress of the transfer. This gives a bored user something to
 watch.  Implies --verbose if it wasn’t already specified.
--stats  This  tells  rsync to print a verbose set of statistics on
 the file transfer, allowing you to tell how effective rsync’s
 delta-transfer algorithm is for your data.

edited Sep 30 '22 at 19:26

Pablo Bianchi

15,657

answered Apr 23 '17 at 12:04

fugitive

1,246

7

This worked for me on macOS. – mango Oct 23 '18 at 20:05
3

What does --stats do specifically? Using just --progress gets me the 98.19M 54% 8.99MB/s 0:00:08 part. – Joshua Pinter Jan 21 '20 at 17:59
3

@JoshuaPinter I believe the statistics you get at the end, i.e. No of files, files transferred, total bytes, etc. are provided by --stats – shubhamgoel27 Mar 15 '21 at 21:07
1

@shubhamgoel27 Thanks for following up. Good to know. – Joshua Pinter Mar 15 '21 at 21:14

A.B. · Accepted Answer · 2015-08-14T20:31:31.673

14

How about this?

rsync_param="-av"
rsync "$rsync_param" a/ b |\
     pv -lep -s $(rsync "$rsync_param"n a/ b | awk 'NF' | wc -l)

$rsync_param

Avoids double input of parameters
$(rsync "$rsync_param"n a/ b | awk 'NF' | wc -l)

Determines the number of steps to complete.
a/ b
1. a/ is the source
2. b is the target

edited Aug 14 '15 at 20:31

answered Apr 15 '15 at 07:46

A.B.

90,397

I accepted this a couple days ago with a big smile. I just un selected and accepted it again though. I do not think I can up vote because I do not have enough credit. Thanks for the assistance btw. – Joseph Smith Apr 21 '15 at 12:53
Any ideas why I get an error, no file or directory found? The command I run is: root@cto:/repo/ubuntu# rsync "$rsync_param" -a --prune-empty-dirs --exclude "*.iso" archive.ubuntu.com/ubuntu/indices/ /repo/ubuntu/indices | pv -lep -s $(rsync "$rsync_param"n archive.ubuntu.com/indices/ /repo/ubuntu/indices | awk 'NF' | wc -l) – Joseph Smith Apr 28 '15 at 12:20
3

"$rsync_param"n is rather strange; the quotes mean it can only work on options with no spaces, and attaching the n to the end means it can only work with short options. Much clearer and easier is simply $rsync_param -n, which specifies dry-run without relying on the format of rsync_param, and by not quoting it, long options could be included as well – Izkata Nov 18 '16 at 05:24
Worked first time, except noticed it doesn’t displays transfer rate – Ciasto piekarz Nov 04 '17 at 16:28

score 4 · Answer 4 · answered Jul 26 '17 at 20:10

4

If your version of rsync does not accept the --info=progress2 option, you can use tqdm:

To install:

pip install tqdm

To use:

$ rsync -av /source /dest | tqdm --unit_scale | wc -l
10.0Mit [00:02, 3.58Mit/s]

answered Jul 26 '17 at 20:10

ostrokach

854
8
11

Gabriel Staples · Answer 5 · 2021-04-18T08:17:58.367

Yeah, do what Jon said: use the --info=progress2 option. But, what do I do if my version of rsync is too old and doesn't support this option? Answer: upgrade rsync!

Here's how to build `rsync` from source on Ubuntu

(tested on Ubuntu 16.04)

Download latest version of rsync: https://download.samba.org/pub/rsync/src/. Ex: "rsync-3.1.3.tar.gz". Save it in a directory WITH NO SPACES AT ALL to ensure it builds right.
In your folder explorer, right-click it and go to "Extract Here".
Enter the extracted folder (ex: "rsync-3.1.3")
Right-click the screen in your folder manager and go to "Open in Terminal." Alternatively, do steps 2 through 4 manually on the command line. Ultimately you just need to be cded into this extracted directory containing the rsync source code.
Check current version of rsync. Make note of this so you can see later it actually got updated.
```
 rsync --version
```

Install necessary tools:

 sudo apt update
 sudo apt install yodl

Build:
```
 ./configure
 make
 sudo make install
```
Ensure it was updated:
```
 rsync --version
```

Sample output:

$ rsync --version

    rsync  version 3.1.3  protocol version 31
    Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    Capabilities:
        64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
        socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
        append, no ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you

    are welcome to redistribute it under certain conditions.  See the GNU
    General Public Licence for details.

Search the man pages for "progress2". You'll now have access to the --info=progress2 option:
```
 man rsync
```

...then press / key and type progress2; press Enter to search for it; press n for the 'n'ext match until you find the entry you're looking for:

There is also a --info=progress2 option that outputs statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify --info=name0) if you want to see how the transfer is doing without scrolling the screen with a lot of names. (You don’t need to specify the --progress option in order to use --info=progress2.)

Also see:

[my answer] "How to use rsync": https://superuser.com/questions/1271882/convert-ntfs-partition-to-ext4-how-to-copy-the-data/1464264#1464264

Partial References:

http://www.beginninglinux.com/home/backup/compile-rsync-from-source-on-ubuntu

score 2 · Answer 6 · edited Apr 28 '15 at 12:31

2

This finally worked:

rsync "$rsync_param" -a --prune-empty-dirs --exclude "*.iso" rsync://archive.ubuntu.com/ubuntu/indices/ /repo/ubuntu/indices | pv -lep -s $(rsync "$rsync_param"n rsync://archive.ubuntu.com/indices/ /repo/ubuntu/indices | awk 'NF' | wc -l)

edited Apr 28 '15 at 12:31

A.B.

90,397

answered Apr 28 '15 at 12:29

Joseph Smith

1,143

score 0 · Answer 7 · answered Jan 12 '21 at 02:08

Something I find useful is using a tqdm like this:

Like @ostrokach said, you can install it either by

pip install tqdm

or

sudo apt install python3-tqdm

Set the params:

SRC=a/
DST=b
PARAMS=-av

And invoke:

rsync ${PARAMS} ${SRC} ${DST} |  tqdm --null --unit-scale --total=$(rsync ${PARAMS}n ${SRC} ${DST} | wc -l)

This will show overall progress counting files (actually - counting lines).

If you want to see the files being copied, you can omit the --null option from tqdm.

How can I view a progress bar when running rsync?

7 Answers7

Here's how to build `rsync` from source on Ubuntu

Also see:

Partial References:

Linked

How can I view a progress bar when running rsync?

7 Answers7

Here's how to build rsync from source on Ubuntu

Also see:

Partial References:

Linked

Here's how to build `rsync` from source on Ubuntu