21

I have millions of images on my ubuntu cloud server. When I move a complete folder containing 12 million images using mv command, it happens almost instantaneously. However, when I mv only images(not the folder) then it takes some time. Is there a way to move all the images as quickly as folders ?

This is what is happening:

  1. src folder has 12 million images and I move this to dst folder using

    $ mv  src ../dst

    Happens immediately

  2. Inside src folder I do this to move: 

    find -maxdepth 1 -name '*.jpg' -exec mv -t ../../dst/ {} +

    This takes some time.

Is there a way to speed up the second process ?

sankit
  • 321
  • 1
    Not a solution - but to clarify: cmd2 must be slower then cmd1 as it is using find and then execs the move for the result. This can never be as fast as a direct move without pre-find-process. – dufte Jun 01 '16 at 07:08
  • probably dst is in a partition whereas ../../dst is on another. – phuclv Jun 01 '16 at 10:53
  • As written this does not even look like a valid find invocation. It's lacking any {} argument where the filename(s) would be expanded. – R.. GitHub STOP HELPING ICE Jun 01 '16 at 20:28
  • I've submitted an edit that alters the title, removing reference to "images" and replacing it with the nub of the matter -- it's moving individual files vs. moving the entire folder. I hope it's accepted by someone with the rep to do it. – Monty Harder Jun 01 '16 at 21:13
  • 1
    It is not a valid invocation of find. find ... -exec mv -t ../../dst/ {} \; would call mv once per file; find ... -exec mv -t ../../dest {} + would be much faster, copying as many files per call as possible, but still not as fast as moving the directory itself as explained by dadexix86. – chepner Jun 02 '16 at 02:35
  • The answers below adress a bit the problem : the 2nd part is making as many time the mv as there are files. To see really why this si so much slower, read http://unix.stackexchange.com/a/169765/27616 (in a nutshell: each "mv" in 2 implies a fork of the "mv" command, just to move 1 file. And it's done for every single files. – Olivier Dulac Jun 02 '16 at 11:34
  • @chepner : I was using the correct find command, I missed {} + while copy pasting. Correcting in the question. – sankit Jul 06 '16 at 11:43

4 Answers4

52

TL;DR: No

For a smaller amount of files, you would not need find but, even in this simplified and smaller case, if you just 

mv *.jpg ../../dst/

it will take more time than moving the whole directory at once.

Why? The point is to understand what mv does.

Briefly speaking, mv moves a number (that identifies a directory, or a file) from an inode (the directory containing it) to another one, and these indices are updated in the journal of the file system or in the FAT (if the file system is implemented in such a way).

If source and destination are on the same file system, there is no actual movement of data, it just changes the position, the point where they are attached to.

So, when you mv one directory, you are doing this operation one time.

But when you move 1 million files, you are doing this operation 1 million times.

To give you a practical example, you have a tree with a many branches. In particular, there is one node to which 1 million branches are attached.
To cut down these branches and move them somewhere else, you can either cut each one of them, so you make 1 million cuts, or you cut just before the node, thus making just one cut (this is the difference between moving the files and the directory).

dadexix86
  • 6,616
  • 4
    You should include that a mv on the same filesystem is just a rewrite the TOC entry. – Videonauth Jun 01 '16 at 07:23
  • I am not sure that I understand what you mean by TOC. As afar as I know, there is no table in ext file systems, or NTFS, or btrfs and so on. FAT has a table (from which it takes the name) but for example ext stores names and blocks, and parents, and children and other informations in the inodes. If you can point me to some reference where it is explained where do ext FS have their TOC and what it is used for, I'll gladly read and update the answer :) – dadexix86 Jun 01 '16 at 07:32
  • Well table of contents or in Ubuntu terms, fs journal. Both has in fact merely the same function, noting somewhere where the information is on the disk. – Videonauth Jun 01 '16 at 07:34
  • 10
    Um. mv *.jpg is likely to fail for 12 million files which is why he uses find. Most Unixes, Linux included I believe (unless someone changed it in the last 5-10 years) have a limited maximum length of the command line. I think it was 64K for Linux for a long time. The same limit applies to environment variables, I am pretty sure. – Zan Lynx Jun 01 '16 at 08:16
  • @ZanLynx I see what you mean. I tried it with few files (I do not have a folder with so many files) and it worked. But it will fail for so many because the * get expanded, am I right? Then would a for loop solve the problem, or is find the best option here? – dadexix86 Jun 01 '16 at 08:33
  • to complete the tree analogy, it requires that all the level of branches are exactly as strong, and therefore as tough to cut and move. – njzk2 Jun 02 '16 at 01:27
  • @dadexix86 both a loop and find will get around the ARG_MAX issue. Of the two, find should be preferred (with -exec mv -t target_dir {} +, note the +) because it will be much faster and run fewer operations. Another alternative would be to use xargs. – terdon Jun 02 '16 at 13:15
  • 1
    Moving a file is more about moving its name. Unix-like directory entries contain a file name and an inode number, which is basically a pointer to the rest of the metadata. A directory is just a special kind of a file. The inode itself doesn't contain the actual data of the file, just pointers to it, so it's a bit misleading to say that anything is moved from an inode. On the other hand, file system journals usually refer to a type of a metadata log mostly used for crash-proofing. – ilkkachu Jun 02 '16 at 19:30
  • 1
    Of course, the terminology is not the point main here. The important bit is exactly what you said: inside a filesystem, a move only needs to touch the metadata. From one filesystem to another, there is no shortcut and all the files need to be moved (recreated) one by one, including their contents. In that case it doesn't matter if one is moving the whole directory or just the files inside, it's going to be about as slow. – ilkkachu Jun 02 '16 at 19:31
  • @ZanLynx: mv -- *.jpg ../../dst/ can work with millions of files in zsh. See Solving “mv: Argument list too long”? – jfs Jun 08 '16 at 12:19
13

It will still be slow because, as noted, the file system has to relink each file name to its new location.

However, you can speed it up from what you have now.

Your find command runs the exec once for each file. So it launches the mv command 12 million times for 12 million files. This can be improved in two ways.

  • Add a plus to the end:
    find -maxdepth 1 -name '*.jpg' -exec mv -t ../../dst/ +
    Check the man-page to make sure it's supported in your version of find. The effect should be to run a series of mv commands with as many filenames as will fit on each command-line.

  • Use find and xargs together.
    find -maxdepth 1 -name '*.jpg' -print0 | xargs -0 mv -t ../../dst/
    The -print0 will use NUL, aka zero bytes to separate the file names. This plus xargs -0 fixes any problems xargs would otherwise have with spaces in file names. The xargs command will read the list of file names from the find command and run the mv command on as many file names as will fit.

Zan Lynx
  • 253
7

Your confusion comes from the file system abstraction which makes you believe that a folder contains files and other folders in a tree-like fashion. This is not actually true: all files and directories within a file system are located on the same level and identified with numbers of some sort, dependent on implementation. Directories are just special files which contain lists of other files.

When you "move" files inside a file system, actual files don't go anywhere. Rather, lists inside directories are updated to reflect the change.

mv src ../dst moves a single list entry from directory . to directory ../dst, so it's fast.

find -maxdepth 1 -name '*.jpg' -exec mv -t ../../dst/ has to move millions of entries, so it's slower. It may potentially be speeded up if you call mv only once and not once per file, and the mv command itself may be optimized to move several directory entries in one step, but there is no way to make it as fast as when you move a single directory.

Dmitry Grigoryev
  • 1,942
4

A Simplified answer

moving a file is done is 3 steps:

  • add() a link to the file to the inode list of the destination folder
  • check if the link was successfully added
  • remove() the link from the list of inodes of source folder if the check above was a success.

this process is the same for a file or a folder.
and obviously doing this for 1 file is 100 faster than doing it for 100 files.

man link is the add()
man unlink is the remove()
mv just uses those two commands above and adds a check in-between to prevent data loss.