9

I have my primary disk completely filled up:

root@kodi:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           385M   12M  374M   3% /run
/dev/sda1        88G   84G     0 100% /
tmpfs           1.9G  4.0K  1.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sdb1       917G  429G  442G  50% /media/Cloud
/dev/sdd1       917G  813G   58G  94% /media/Tera
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           385M     0  385M   0% /run/user/1000
root@kodi:/#

/dev/sda1 is my primary disk where ubuntu is installed:

root@kodi:/home/fmf# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=9fab4895-7ccb-4415-b26d-311a17036cda       /               ext4    errors=remount-ro 0       1

# swap was on /dev/sda5 during installation
UUID=7b158f58-e4c1-4717-aa5f-dbeaa79ab93c       none            swap    sw              0       0

UUID=f9079f52-7661-48ad-9bc4-0d2452be66af       /media/Tera     ext2    defaults,nofail 0       0
UUID=fb1f92ee-54f5-44f8-ba92-544e90e6dfeb       /media/Cloud    ext2    defaults,nofail 0       0

root@kodi:/home/fmf#

Did some google search and found some commands to test who is the responsible, but I can't figure out who is filling it:

root@kodi:/# du --exclude=/media -cksh * | sort -hr | head -n 15
du: cannot access 'proc/16360/task/16360/fd/3': No such file or directory
du: cannot access 'proc/16360/task/16360/fdinfo/3': No such file or directory
du: cannot access 'proc/16360/fd/3': No such file or directory
du: cannot access 'proc/16360/fdinfo/3': No such file or directory
1.3T    total
1.3T    media
3.5G    home
3.0G    usr
1010M   var
645M    root
630M    lib
99M     boot
17M     bin
15M     sbin
13M     etc
12M     run
196K    tmp
16K     lost+found
12K     srv
root@kodi:/#

Apparently there isn't any file or directory so big to fill the 84G of my disk.

Some days ago I found that the disk was full due to .xsession-errors that went crazy. I found this is a known bug in ubuntu and I solved creating a crontab line that every ten minutes delete .xsession-errors. In fact right now I don't have it anymore in my home directory.

Any help, please?

effemmeffe
  • 583
  • I don't have .xsession-errors, the cronjob deleted it. I don't get what do you mean with official releses of Ubuntu: I have Ubuntu 16.04.1 LTS. – effemmeffe Jan 30 '17 at 09:17
  • 2
    To see if there are deleted files still accessed by any process, sudo lsof | grep deleted will give hints. – ridgy Jan 30 '17 at 09:54
  • @ridgy's comment is spot on. If your .xsession-errors issue is at fault, that will highlight it; if not, it's still very likely to point you in the right direction. – user Jan 30 '17 at 10:46
  • 4
    @effemmeffe In UNIX, deleting a file does not actually delete it until the last handle to it is closed (this is why you can always delete a file in UNIX, where in Windows you'd get "access denied" errors etc). So the .xsession-errors file is still there, filling up your disk space, because whatever is logging the errors keeps the file open. The best way to fix this properly is to figure out what causes the errors and fix it. – Muzer Jan 30 '17 at 13:27
  • 1
    If the issue is with open file handles on deleted files, rebooting the system should "magically give you back" all the missing space. Might be a useful troubleshooting step. – Floris Jan 30 '17 at 18:50

3 Answers3

19

The problem with your cronjob is that .xsession_errors is most likely still opened by some applications or system services which is why it will be hidden from the filesystem table when deleted, but it still is on disk and still errors will be written to it.
So it will fill up the disk, but now you can't see it anymore.

@rinzwind aims for exactly this behaviour when he (correctly) suggests to remove the cronjob and look for the errors. This is the only way to fix this problem properly.

As a workaround you could truncate the .xsession_errors file with a cronjob like this:

17 */2 * * *  truncate -cs 0 path/to/.xsession_errors

But before doing this you REALLY should try to fix the underlying problems that create those error messages in .xsession_errors

terdon
  • 100,812
Phillip -Zyan K Lee- Stockmann
  • 2,353
  • @terdon I like /etc/crontab more myself :=D – Rinzwind Jan 30 '17 at 10:40
  • 1
    @Rinzwind not for modifying files in a user's home directory though. At least run it as the user and not as root! – terdon Jan 30 '17 at 10:59
  • @terdon, @rinzwind you are both right: I should have paid attention. having this script run as user root would be careless and could create some other problems. – Phillip -Zyan K Lee- Stockmann Jan 30 '17 at 11:00
  • While you were answering (note to myself: find a way to be notified immediatly when a new answer is posted) I found the same answer. I'd like to follow the path from @Rinzwind answer so I decided to rotate the file based on its size. Still no luck with that, but I'll try again and it seems to me it would be a good compromise between losing everything in the log and having the system unusable due to disk space. – effemmeffe Jan 30 '17 at 13:23
  • By the way, I found that the error is due to a problem with the media player kodi and python cryptograpy that is already solved with the new version of the package, now I just have to learn how to install a package from 17.04 to my 16.04 version. Meanwhile I'll rotate the file or, if I can't make it work or if someone will point it's not a good solution, I'll truncate it. Thanks. – effemmeffe Jan 30 '17 at 13:23
  • 2
    rotation has the same problem as with deletion: kodi will not recognize the file is rotated and will continue to write in there, until you notify it to reopen this file. (most likely there is no such thing and you have to restart kodi) – Phillip -Zyan K Lee- Stockmann Jan 30 '17 at 14:15
  • 1
    @terdon truncate -c won't create a file and it won't change the ownership of the existing file. It's good not to run it as root, but ownership of the file isn't a reason why. – hobbs Jan 30 '17 at 23:30
  • @hobbs I stand corrected, you are absolutely right. I basically ignored the -c which, as I just found out, means "do not create any files". And you're also right about not changing ownership which is even more surprising. I guess I'll have to strace it and figure out what is actually happening. Anyway, thanks for the correction! – terdon Jan 31 '17 at 22:32
10

Who is filling my disk?

since you do this ...

I found this is a known bug in ubuntu and I solved creating a crontab line that every ten minutes delete .xsession-errors

it is impossible to answer this question.

Please follow the following to get us the errors we need to see:

  • Remove that crontab line you added that removes .xsessions_errors.
  • Reboot and then check from command line what is filling up .xsessions_errors using tail -f 100 ~/.xsessions_errors.
  • When you find any problems that are repeated a lot copy some of them into your question.
  • Add the crontab line back so you can use your system.
  • Wait for a fix for the problem and apply then. Then remove the crontab line again (deleting the .xsessions_errors file should always be avoided).
Rinzwind
  • 299,756
  • 7
    "When you find any problems that are repeated a lot copy some of them into your question." No, that would be a new, different question. – Lightness Races in Orbit Jan 30 '17 at 15:16
  • Follow-up: I removed the crontab and now the .xsession-errors file is free to grow. But it's not. And my disk space is still dropping. So the problem wasn't there. A reboot stops the disk eating, but I'd llike not to reboot the machine every day. Any clue? – effemmeffe Jan 31 '17 at 11:20
3

When there is a big differences (say, more than a few percent) between (as root) df -g /some/partition_root and du -gx /some/partition_root ( -x tells du to limit itself to the same partition, usefull to check '/' for example) : there are probably file(s) still opened that some processes are still writing to, but that you deleted on disk (hence: the file still exists as long as it is opened by the processes, and fills up the disk space (df -g sees that) but as there are no longer links (or filenames) to its inodes, du -gx misses them.

In your case, compare the outputs of :

df -g /
du -gx /

To Find out which are the files with less than 1 links (ie, deleted files but still opened) :

lsof -nP +L1  /

Check the process names and the pids to see which are the processes writing to those "ghost" files, and act accordingly (some you may be able to stop/start, and when they are stopped the file will be released and its space freed, others may necessitate a proper reboot)

Olivier Dulac
  • 996