Browsing folders with more than 100000 images

Question

I have a folder with 322000 images in it. When I go to that folder with any file manager it gets stuck in loading.

How can I go there and view or delete images?

Answer 1

Automatically divide your files into a (recursive) directory with an arbitrary number of files per (sub) folder / folders per superior folder

The easiest and IMHO most efficient way is to have a script reorganize the files into folders, if necessary even different layers of directory levels. This will make your files browsable without choking nautilus.

The script below will do that for you. It will create folders with an arbitrary number of files. These folders will be organized into sub folders if they exceed an (the same) arbitrary number, etc. In other words; each (sub-) level will have the same maximum number of files / sub directories, making browsing easily possible.

Each of the created folders shows the folder number + the number of created sub levels (where e.g. 22_1 only contains files):

The test

I tested in on a directory of 300.000 files, to be reorganized in chunks of (max) 100 files, to be organized into superior directories of (max) 100 folders etc.
It took less then a minute on my system. A test of 100.000 files into smaller chunks was a matter of seconds.

The script

#!/usr/bin/env python3
import subprocess
import os
import shutil

#--- set the directory to reorganize below
dr = "/path/to/directory"
#--- set the number of files/folders per level
size = 100

level = 0
def move(fn, drn, level):
    folder = dr+"/"+str(drn)+"_"+str(level)
    if not os.path.exists(folder):
        os.mkdir(folder)
    shutil.move(dr+"/"+f, folder+"/"+f)

while len(os.listdir(dr)) > size:
    level += 1
    fn = 0; drn = 1
    for f in os.listdir(dr):
        if fn < size:
            move(fn, drn, level)
        else:
            fn = 0
            drn += 1
            move(fn, drn, level)
        fn += 1

How to use

1. Copy the script into an empty folder, save it as reorganize.py
2. In the head section, set the path to your directory, the desired number of files per subdirectory (= equal to folders per containing directory).

3. Run it by the command:

   python3 /path/to/reorganize.py
   

Note

The script (as it is) just creates a directory structure where each level has a defined number of files/folders. It does not take into account any kind of organisation by name, date or whatever.

---

EDIT

As requested in a comment, a script to move the files back into one flat directory after having processed the files.
The usage is pretty much the same. As directory, set the same directory as the first script, but that seems obvious.

#!/usr/bin/env python3
import shutil
import os

#--- set the directory, the same as the first script
dr = "/path/to/directory"
#---

# move the files back
for root, dirs, files in os.walk(dr):
    for file in files:
        shutil.move(root+"/"+file, dr+"/"+file)
# remove the (now empty) subdirectories
for d in os.listdir(dr):
    folder = dr+"/"+d
    if os.path.isdir(folder):
        shutil.rmtree(folder)

Note

As mentioned in a comment, the script assumes there is not risk of name clashes, since all files initially came from the same (flat) directory.

Answer 2

This is a problem in the file manager nautilus when working with directories containing large number of files, this is described in this bug.

I advise you in this case to use some fast File manager. you can use gnome-commander it's 5X to 6X times faster than nautilus and this ratio gets bigger when dealing with large number of files. It's not that pretty GUI but it should help you to fix your problem.

To install it:

sudo apt-get install gnome-commander

Now open the directory with large number of files:

gnome-commander /path-to-dir

Answer 3

I tried Thunar file manager with images in the order of 10,000. Displays thumbnails fairly quickly on local drives and opens network drives too but much slower. I have not tried it for a larger number of files, though. But at least, it does a much better job than Nautilus without compromising the graphical interface

sudo apt-get install thunar

There is also a script to make it your default File Manager (if you like it). You can find the script here:

https://help.ubuntu.com/community/DefaultFileManager

Answer 4

I don't have any personal experience with libraries that big, but I've heard good about dolphin.

Dolphin is the file manager used in KDE distributions such as Kubuntu. You can install it and run it just fine on the Ubuntu Desktop, although the footprint might be quite large, as it has to download and install a lot of KDE dependencies.

Download Dolphin File Manager

Another approach might be to use a photomanager. I would assume such tools would be optimized to display large collections.

For photomanagers there are quite a few alternatives, but a couple of popular ones are:

Shotwell, Digikam and F-Spot

Answer 5

You could go through terminal to that folder , do gnome-open image.jpg or xdg-open image.jpg , which should open one of the images with gnome-image-viewer or more commonly known as eye of gnome(eog). Next, if you hit F9 or go to View -> Image Gallery you can preview thumbnails.