2

I have a problem with a server running Ubuntu 16.04.05. After running a python script who created more than 65 million of files I have an error message saying that I have no space left:

12 zip -r 31.zip 31
zip I/O error: No space left on device
zip error: Could not create output file (31.zip)

I found some similar problems online: No space left on the device though there is space and No space left on device even though there is, but I think I have a different one. The solutions posted in these two answers are not working for me.

The following is the structure of the storage:

➜  12 df -k
Filesystem      1K-blocks      Used Available Use% Mounted on
udev             32963768         0  32963768   0% /dev
tmpfs             6597356      9244   6588112   1% /run
/dev/sda1        32895856  26382068   4819736  85% /
tmpfs            32935580        96  32935484   1% /dev/shm
tmpfs                5120         0      5120   0% /run/lock
tmpfs            32935580         0  32935580   0% /sys/fs/cgroup
/dev/sdb       1031992064 664288464 322578204  68% /media/hdd1
tmpfs             6587116        20   6587096   1% /run/user/118
tmpfs             6587116         0   6587116   0% /run/user/1001

I am working and trying to zip a folder in /media/hdd1 which should have plenty of space left. The dimension of the folder I am trying to zip is:

➜  12 du -sh 31
49M     31

EDIT:

➜  12 df -i
Filesystem       Inodes    IUsed   IFree IUse% Mounted on
udev            8240942      478 8240464    1% /dev
tmpfs           8246695      617 8246078    1% /run
/dev/sda1       2097152   500872 1596280   24% /
tmpfs           8233895        5 8233890    1% /dev/shm
tmpfs           8233895        6 8233889    1% /run/lock
tmpfs           8233895       16 8233879    1% /sys/fs/cgroup
/dev/sdb       65536000 65536000       0  100% /media/hdd1
tmpfs           8246695       15 8246680    1% /run/user/118
tmpfs           8246695        4 8246691    1% /run/user/1001
Guido Muscioni
  • 123

2 Answers2

4

"After running a python script who created more than 65 million of files" here is your problem.

The ext4 file system Ubuntu uses has only a limited number of so-called "inodes", which is determined when you format a partition. You need at least one inode per file or directory on the file system.

Now if you create huge numbers of small files, you'll exhaust the available inodes faster than the available storage space. Both equally result in the same symptoms of a seemingly full disk though.

You will need to get rid of those tons of tiny files and e.g. put them on a different drive, in a zip file, or delete them altogether.

Unfortunately the number of inodes can not be changed dynamically, as far as I know. If you really need more on your current partition, you'll probably need to back up your data and reformat everything.

Related:

Byte Commander
  • 107,489
  • As there is still space, can I create a partition with more inodes on it? The python script returned correctly so I am wondering if the number of inodes was equal to the generated number of files (which I think, has a probability of 0). – Guido Muscioni Feb 12 '19 at 23:31
  • 1
    Sure, if you have remaining unpartitioned space, you can make an additional partition there, possibly with increased inode ratio. Free space on your current partition is not suitable to create a new partition though. You could maybe shrink your current partition to make free unpartitioned space, but I don't know if/what effects shrinking might have on a file system exhausted of inodes. What you could also do instead might be to create a "file system image file" on your disk. Basically one large file that contains a whole virtual file system, which can be mounted separately. – Byte Commander Feb 12 '19 at 23:44
1

As ByteCommander explained, you have a problem with inodes.

Another way to takle this problem is to look for the millions of little files that are consuming the inodes.

sudo du -a -x -d 1 --inodes /media/hdd1 | sort -nr | head -20

will show you the directories consuming the most inodes on /media/hdd1 - you can then travel down through the directories until you find one filled with a bunch of files.

Charles Green
  • 21,339
  • Well, I know these files are not trash, I wrote the code. The point is if I can fix the problem without deleting the files. Otherwise I will change the code. – Guido Muscioni Feb 13 '19 at 00:17
  • @GuidoMuscioni Sorry for the implication - I will modify my post. Why do you need to create 65 M files? – Charles Green Feb 13 '19 at 00:18
  • No problem, at a first glance is fine to think they are trash. It is a long story, but basically, I am dividing a set of song listened by a set of user. I am saving them base on year/month/day/hour_user.csv. I have more than 200000 users and 18 years, thus a maximum of: 20000036518*24=31536000000 files. In reality these files are less, but that the worst case scenario. – Guido Muscioni Feb 13 '19 at 00:47
  • @GuidoMuscioni I'm assuming you are doing this for some analysis purpose in the future - but wouldn't this information really be more readily accessed in a SQL database? – Charles Green Feb 13 '19 at 00:49
  • @GuidoMuscioni And you do know that your potential number of files exceeds the theoretical number of files that an ext4 system can handle https://serverfault.com/a/104989 – Charles Green Feb 13 '19 at 00:59
  • Yes, I am reading. I thought about SQL, but the raw format of these files does not directly map on SQL import types. Furthermore, SQL is not really suited for temporal datasets (temporal joins can be extremely slow) and every time materializing a view for a single hour is simply not feasible. I thought about Cassandra, which has support to temporal indexed datasets, but, I do not know how to use it. Anyway, it is for analysis purposes and I will need to compute this structure only once. – Guido Muscioni Feb 13 '19 at 01:07
  • There's an interesting query here discussion datetime indicies - since you are rounding to hours anyhow, the tables associated with date and time should be much smaller... – Charles Green Feb 13 '19 at 01:28
  • I am fixing it writing a join routine after every 1000 created files, in this way, I will end up with 3651824 = 157680. Totally fine with ext4. – Guido Muscioni Feb 13 '19 at 01:36
  • Let us continue this discussion in chat. – Guido Muscioni Feb 13 '19 at 02:03