No more space on disk

Question

I'm using Ubuntu 14.04. For some time I haven't been able to run more programs simultaneously. This is what df -h gives:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        29G   25G  3.4G  88% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             16G  4.0K   16G   1% /dev
tmpfs           3.2G  724K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             16G   59M   16G   1% /run/shm
none            100M   40K  100M   1% /run/user
/dev/sda2       438G  138M  416G   1% /scratch
AFS             8.6G     0  8.6G   0% /afs

and of df -i:

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/sda1       1875968 1875968        0  100% /
none            4106915      11  4106904    1% /sys/fs/cgroup
udev            4103190     499  4102691    1% /dev
tmpfs           4106915     524  4106391    1% /run
none            4106915       3  4106912    1% /run/lock
none            4106915     122  4106793    1% /run/shm
none            4106915      27  4106888    1% /run/user
/dev/sda2      29138944     709 29138235    1% /scratch
AFS             9000000       0  9000000    0% /afs

I have also tried the suggestion here: bash: echo: write error: No space left on device , does not help, actually it makes it worse. Now, if I enter the terminal and partially write the name of a folder ,I press Tab and get:

cd Pybash: cannot create temp file for here-document: No space left on device

where I just wanted to write

cd Python

Edit: The output of sudo du -hxd1 / is :

1.5G    /var
22G /usr
18M /etc
16K /lost+found
36K /home
349M    /opt
68K /tmp
9.7M    /bin
4.2M    /libx32
12K /.config
2.5M    /root
4.0K    /mnt
4.0K    /lib64
13M /sbin
81M /boot
505M    /lib
3.5M    /lib32
12K /media
4.0K    /srv
25G /

Answer 1

Space is not the issue here. You have a directory with very many empty (or near-empty) files. Each of them takes an inode, and you have used them all. Find that directory:

for i in /* ; do echo "$i" ; find "$i" | wc -l ; done

(You should do this as root). This will probably list one directory with a large number of files. Drill down until you find the files that have depleted your inodes. Delete all those files. Then you'll be OK for the moment. Where these files come from is, of course, another story.

Answer 2

As you can see from df -h, you still have space left on your root partition. However, df -i shows you have run out of inodes. On my computer, I have 7.3G used on my root partition, but only 320,000 inodes used, so you probably have a lot of very small files clogging things up somewhere.

For a discussion of where, see, e.g. Stack Exchange - Find where inodes are being used. I would try the following from that page:

sudo find / -xdev -printf '%h\0' | sort -z | uniq -cz | sort -nrzk 1 | tr '\0' '\n' | head -n 50

If you are on a more recent version of Ubuntu (e.g. 15.10), then this shorter command might also be faster:

sudo du / --inodes -xS | sort -rh | head -n 50

Either will give you a list of directories ordered by how many inodes they are using. You will probably find one with a very high number (likely to be in /var somewhere). This should give you an idea of which files you can delete, but if you have any questions, do ask back here first.

As a reference for what might be normal, on my computer (running Ubuntu 15.10), I get the following output from that command:

3297    /usr/share/app-install/desktop
3022    /usr/share/man/man3
2325    /usr/bin
2149    /usr/share/man/man1
2104    /usr/share/app-install/icons
1816    /usr/lib/x86_64-linux-gnu
1089    /usr/src/linux-headers-4.2.0-19/include/linux
1089    /usr/src/linux-headers-4.2.0-18/include/linux
1089    /usr/src/linux-headers-4.2.0-16/include/linux
...

---

Here is an explanation of the simpler command. sudo is needed to run the du command as root, so we can find files in folders your normal user might not be able to read. du / gives the Disk Usage of the root directory /; --inodes tells it to list inode usage, -S to only list files in the directory itself and not in subdirectories, and -x to not cross filesystem boundaries (as your problem is definitely on / and not on, for example, /scratch). Then we pipe the output of that command to sort to find the worst offenders; -r reverses the sort so the biggest are at the top, and -h sort numerically and not alphabetically. Finally, head -n 50 only outputs the first 50 lines (i.e. the worst 50 offenders).

---

Here is an explanation of the more complicated command. sudo is needed to run the find command as root, so we can find files in folders your normal user might not be able to read. find / gives a list of all files on the root directory /; -xdev tells find not to cross filesystem boundaries (as your problem is definitely on / and not on, for example, /scratch); -printf '%h\0' prints the directory name ending with a zero byte. Then we pipe the output of that command to sort -z and then uniq -cz to count how many times each directory was printed (i.e. how many files are in each directory); sort is needed here since uniq needs a sorted list. Then we sort to find the worst offenders; -r reverses the sort so the biggest are at the top, and -n sort numerically and not alphabetically. tr '\0' '\n' replaces zero bytes with new lines. Finally, head -n 50 only outputs the first 50 lines (i.e. the worst 50 offenders).