0

How can I copy around 1000 subfolders each having 4000 images into one single folder named all_images?

I tried the following:

$ cp upload/*/* all_images/

and got the following error:

bash: /usr/bin/cp: Argument list too long

Please note that I tried this on two small subfolders and it worked.

Mona Jalal
  • 4,545
  • 21
  • 68
  • 99
  • 2
    You should also be aware of the filesystem performance hit with a directory with 4 million files. – ubfan1 Apr 19 '19 at 22:44
  • @ubfan1 what are our alternatives? should we just live them be at their current structure and modify our python code to go through each of these subdirectories to read images? – Mona Jalal Apr 19 '19 at 22:45
  • @ubfan1 additionally, is there a link that talks about performance hit with 4M files in one directory? why is it not hit if these 4M are distributed in 1000 directories? – Mona Jalal Apr 19 '19 at 22:46
  • 1
    See https://serverfault.com/questions/43133/filesystem-large-number-of-files-in-a-single-directory Filesystems may be tweaked at creation for files (inodes etc.) but just because you can do it, doesn't mean you'd be happy with the result.Try tests on a directory with 30,000 and 60,000 files, that's a usual practical cutoff. – ubfan1 Apr 19 '19 at 23:02
  • 1
    Also see https://askubuntu.com/questions/584315/placing-many-10-million-files-in-one-folder for some issues and suggested alternatives. – ubfan1 Apr 19 '19 at 23:17

0 Answers0