Directory using more space than expected

Question

I have a directory on Ubuntu that contains 262144 files. Each file is a 25x25 png image. On average these files are approximately 1.14kb and no file is larger than 2kb. Yet this directory is using up 3.1GB of disk space. How is this possible? By my calculations this directory should be using 262144 * 1140 = 298844160 bytes, which is only 0.298844 GB.

Here are the steps I followed to get this information.

I ran ls -1 -f | wc -l to count the number of files in the directory. This returns 262146 (i.e., 262144 + 1 + 1 for . and ..).

Next I ran find . -size +2k and the result was just ..

Finally I ran du -sh culprit_directory and the result shows 3.1G culprit_directory.

There are two things that I imagine could be happening:

1. Ubuntu needs extra space to store a directory that contains a very large number of very small files. Possible, but a whole order of magnitude?
2. I am making a mistake in my calculations and this is the expected size for the directory. Also possible, but I am unable to see where I made this mistake.

If anyone with more experience with Ubuntu's internal file storage could advise me I would greatly appreciate it.

EDIT: I have added one of the png files. This one is 591 bytes in size.

EDIT:

Thanks to muru's helpful comments below, I have determined that each file is actually using 12KB on disk, even if it only consists of a few hundred bytes. Using the new numbers, we get 262144 * 12000 = 3145728000, which gives us 3.145728GB.

I guess my new question would be how to avoid each file using so much space?

Answer 1

Answering the follow-up question, probably the easiest thing to do in these cases is to create a small ad-hoc filesystem and loop-mount it. Something like this:

$ dd if=/dev/zero of=imgdisk.img bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 0.425628 s, 1.3 GB/s
$ du -h imgdisk.img 
513M    imgdisk.img
$ mkfs.ext4 -b 2048 imgdisk.img 
mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: done                            
Creating filesystem with 262144 2k blocks and 32768 inodes
Filesystem UUID: 8837a733-6b75-4326-bb72-9372538653ad
Superblock backups stored on blocks: 
        16384, 49152, 81920, 114688, 147456

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

$ mkdir imgmount
$ sudo mount imgdisk.img imgmount -o loop
$ ls imgmount/
lost+found

copy the images in there (sized as it is it might be just too small for your files; make the 512 a 513 if so), umount the loop filesystem, mount it over the directory with all the images. If that works, umount it from there, delete the original images, edit /etc/fstab so it mounts the loopback filesystem in the right place, mount -a and you're set.

Edit: you can use -b 1024 instead of 2048, also.

Answer 2

Different pieces of software report different disk space usage statistics in 2 ways. Files can have a size that is the number of bytes in a file, and a "physical size" which is the sum of the cluster sizes used by that file. The physical size, or "cluster size" is the minimum chunk the OS keeps track of when referring to space used on a disk. So if you have one file containing 1 byte, and your cluster size is 8 kilobytes, the file uses a minimum of 1 cluster, or 8kb.

Wasted disk space really isn't normally a problem, even as disk sizes increase and clusters sizes increase to 32kb or 64kb.

 guess my new question would be how to avoid each file using so much space?

Put files that are used less into an archive file, like a .zip file. The OS has a limit on how many clusters it can keep track of.

Maybe someone else can explain the limit on cluster sizes for several OSs.