How to check the progress of md5sum on many huge files?

Question

I have some 500 files of 650MB each, named bigfile.000 to bigfile.199 and I need to calculate the MD5 checksum of all files combined. The command I am using is:

cat bigfile.* | md5sum

I have the time, and I understand it will take hours to complete, but I still want to check the progress. Is there any way to do so? Like piping something in between and do a word count that doesn't affect performance much?

I think it's not a duplicate question - they are different questions but simply the same answer that solves two different problems. cp is different from cat anyway. — Kenneth L, May 12 '14 at 06:45

score 37 · Accepted Answer · edited May 18 '14 at 17:07

37

Use pv. It is in the repos, and shows a nice progress bar.

pv bigfile.* | md5sum

should do what you want.

edited May 18 '14 at 17:07

Seth

58,122

answered May 09 '14 at 11:20

Jos

29,224

score 5 · Answer 2 · answered Dec 07 '20 at 19:04

5

Progress bar plus stats: ETA, ...:

pv -pteIrabT file | md5sum

For example:

# pv -pteIrabT hugefile.tar.lzo | sha1sum 
7.59GiB {----} 0:03:22 [39.1MiB/s] [38.5MiB/s] [>                          ]  7% ETA 0:41:47 ETA 19:45:26

answered Dec 07 '20 at 19:04

Pablo Catalina

51

1

This doesn't seem to give any more output than just using pv without any options. – SiHa Jun 29 '21 at 10:35

How to check the progress of md5sum on many huge files?

2 Answers2