disk is full, but cannot find big files or folders

Question

Ubuntu server shows me that I use almost all disk:

Usage of /:   95.5% of 118.12GB

And I try to find big folders and files, run ncdu:

ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                                 
--- / ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    5.5GiB [##########] /root                                                                                                                                                                               
    2.3GiB [####      ] /var
  628.6MiB [#         ] /usr
  209.9MiB [          ] /lib
   28.2MiB [          ] /boot
    8.6MiB [          ] /bin
    7.7MiB [          ] /sbin
    6.6MiB [          ] /etc
  208.0KiB [          ] /run
  112.0KiB [          ] /tmp
   48.0KiB [          ] /opt
e  16.0KiB [          ] /lost+found
    8.0KiB [          ] /dev
    8.0KiB [          ] /media
    4.0KiB [          ] /lib64
e   4.0KiB [          ] /srv
e   4.0KiB [          ] /selinux
e   4.0KiB [          ] /mnt
e   4.0KiB [          ] /home
    0.0  B [          ] /proc
    0.0  B [          ] /sys
@   0.0  B [          ]  initrd.img
@   0.0  B [          ]  vmlinuz

According to ncdu I use about 10 GiB of 128 GiB - it is about 10 %. Contradiction.

How to clean my ubutntu server without rebooting?

I thought that ncdu lies and used another apps to find big files and folders. All of them shows the same result as ncdu.

And df -h command shows that disk is full.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda       119G  113G     0 100% /
udev            2.0G  8.0K  2.0G   1% /dev
tmpfs           788M  212K  788M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm

Update

sudo du -sch /* result:

$ sudo du -sch
8.7M    /bin
29M /boot
8.0K    /dev
6.6M    /etc
4.0K    /home
0   /initrd.img
210M    /lib
4.0K    /lib64
16K /lost+found
8.0K    /media
4.0K    /mnt
48K /opt
du: cannot access /proc/4470/task/4470/fd/4': No such file or directory du: cannot access/proc/4470/task/4470/fdinfo/4': No such file or directory
du: cannot access /proc/4470/fd/4': No such file or directory du: cannot access/proc/4470/fdinfo/4': No such file or directory
0   /proc
5.0G    /root
212K    /run
7.8M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
112K    /tmp
629M    /usr
2.3G    /var
0   /vmlinuz
8.1G    total

8.1G total as usual. But I see cannot access rows, maybe problem because of them.

Then I checked the biggest folder in /. It is /root :

$ sudo du -sch /root
96K /root/Downloads
2.5G    /root/Dropbox
36K /root/nohup.out
4.0K    /root/npm-debug.log
4.0K    /root/readonly
980K    /root/redis-2.6.16.tar.gz
228M    /root/tmp
2.7G    total

Answer 1

I was running into this same issue on our lab machines and using this command

du -sch .[!.]* * |sort -h

I was able to find hidden files like inside of users trash bins that they had yet to delete.

Credit to here where I originally found this answer.

Answer 2

Check for deleted files that are still being held open by a process:

sudo lsof | grep deleted | less

That'll show the pid and file descriptor. I had this exact problem on a server, nothing in ncdu but disk filling up. It turned out to be a nightly process that moved files to a mounted samba share and occasionally didn't close out the file handle correctly, it seems.

If you find deleted files and want to clean them up, a reboot is probably easiest if that's acceptable. Or you can try killing the process. Or if you're sure they're not being used, you can manually empty them out, with something like this:
> /proc/14487/fd/12

Answer 3

We had this same issue and it turned out to be docker images, stored under var/lib/docker

ncdu does not list these as they are not visible to users. even running ncdu under sudo does not help.

This command purges all existing docker images...

docker rmi $(docker images -a -q)

Answer 4

Following command will show disk utilization for /home directory with --max-depth=1:

sudo du -h -d 1 /

From manual:

-d, --max-depth=N print the total for a directory (or file, with --all) only if it is N or fewer levels below the command line argument; --max-depth=0 is the same as --summarize.

Answer 5

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

please refer: https://serverfault.com/questions/315181/df-says-disk-is-full-but-it-is-not

Answer 6

Be sure to check your disk mounts. None of the solutions I've seen here can identify space being taken up by a folder which has a mount placed over it.

Answer 7

One reason for "hidden" big files is a possibility to mount a folder (to some other folder/resource residing on a different disk) - the whole content of the original folder is kind of "hidden" - "du" won't show it, because it will be "overlapped" with the mounted resource.

Answer 8

I was experiencing this issue myself. Disk full, completely unable to locate the offending directories and/or files. However, I did solve this puzzle.

All of the previous answers are extremely helpful and legitimate but if you are, like me, unable to find the offending directories and/or files using the tips above, I'd like to share my solution.

I have an external USB drive (ex: dev/sdeX) that I used exclusively for backing up the content of root (ex: /dev/sdaX). Using that as a clue, I unmounted all mount points, excluding root and other required mount points. From there, I noticed the mount point to which the external USB drive mounts to is utilizing over 150G of space. Sure enough, I venture to this directory and see that it contains the same exact file structure and content as the external USB drive (when mounted). I mounted my external USB drive to verify backup data is intact (it is, check) then unmounted it and deleted the contents of that directory (for peace of mind, make sure to verify the mount point has been unmounted). Once I did that, usage on / (/dev/sdaX) went from 100% usage to 13%, which is right about what I expect.

Point of my answer... try unmounting all external drives and double check the mount point for data that should be on the external drive, not your root drive (/dev/sdaX).

Hope this helps.

Answer 9

You can run the command below to find the top 10 files by disk usage:

find / -type f -printf '%s %p\n' 2>&1 \
     | grep -v 'Permission denied' \
     | sort -nr \
     | head -10

Answer 10

switch to root user than run ncud on required folder, it will work. due to permission issue it was not showing files taking space. It's in my case, maybe something else works for you.