10

Please see Edit #3 if you're looking for a solution.

I'm running Xubuntu 16.04 LTS, and I have a ~40 GB root partition which is 100% full according to System Monitor (as root). It's definitely full, since many programs aren't functioning correctly.

System Monitor Disk Usage

However, for some reason, I can't seem to figure out what is using the space! Baobab (as root) only reports a total of 15.5 GB used on my root partition!

Baobab Disk Usage

EDIT: Also, here's /var - people said that it's big. Baobab only reports 1 GB for /var, and /var/log is empty. I've tried running sudo rm -R /var/log and there was no effect.

enter image description here

So, how do I find out what is using my disk space, and how do I prevent it from filling up my root partition? This is a huge problem, please help! Thank you in advance :)

EDIT 2: As posted in the answer section, sudo lsof / | awk '{if(\$7 > 1048576) print \$7/1048576 \"MB\" \" \" \$9 }' | sort -n -u returns 11222.7MB /var/log/kern.log 11222.9MB /var/log/syslog, however, I can't seem to figure out how to delete these files, and additionally, I would like to figure out how I can permanently prevent these files from growing this large. This answer to another question suggested that I look into the logs and see what's filling them up, so ideally I'd like some way to read the contents of these mystery files.

EDIT 3: I have temporarily fixed this issue by mounting /var/log on a separate partition.

However, there is still some kind of bug that's causing this. Please, if you want this bug to be fixed, please bring information (or at least give attention) to the bug reports: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1643719 and https://bugzilla.kernel.org/show_bug.cgi?id=188331 Thank you :)

Random Person
  • 1,496
Aaron Franke
  • 1,126
  • You could use tail command as in the answer you have mentioned. – user.dz Nov 21 '16 at 21:05
  • tail: cannot open '/var/log/syslog' for reading: No such file or directory – Aaron Franke Nov 21 '16 at 21:38
  • You need to run it with sudo – user.dz Nov 21 '16 at 21:51
  • As root, do cd /proc/$(pidof rsyslogd)/fd ; ls -l | grep /var/log/kern.log. Look at the number to the left of the -> . Run tail -200 thatnumber. Fix the problem (could be a bad piece of hardware). – Mark Plotnick Nov 21 '16 at 21:52
  • Rebuilding the /var/log hierarchy that you removed will be harder, as there are a few dozen files and directories there that need to have specific owners and permissions. Do you have backups? – Mark Plotnick Nov 21 '16 at 21:53
  • Nov 21 14:52:02 aaron-xub16desk kernel: [67732.234857] ACPI Exception: AE_NOT_FOUND, while evaluating GPE method [_L6F] (20150930/evgpe-592) Nov 21 14:52:02 aaron-xub16desk kernel: [67732.235872] ACPI Error: [PGRT] Namespace lookup failure, AE_NOT_FOUND (20150930/psargs-359) Nov 21 14:52:02 aaron-xub16desk kernel: [67732.235874] ACPI Error: Method parse/execution failed [\_GPE._L6F] (Node ffff8804558d8000), AE_NOT_FOUND (20150930/psparse-542) – Aaron Franke Nov 21 '16 at 22:27
  • I do not have backups. Also, I haven't tried rebooting yet, because I was hoping to read the contents of the logs to see what's causing this issue, because I'd really like a solution other than mounting /var/log on another partition (which is all that I can think of). – Aaron Franke Nov 21 '16 at 22:29
  • You can cpthat numeric file over to some place like /storage/kern.log.backup to preserve it. Please post the most common repeated lines in it as a new question. A google search found this, which may help get rid of the flurry of _L6F error messages: http://jhshi.me/2015/11/14/acpi-error-method-parseexecution-failed-_gpe_l6f/index.html – Mark Plotnick Nov 21 '16 at 22:35
  • It appears to be mostly ACPI errors. I'm going to try this https://unix.stackexchange.com/questions/92366/acpi-errors-exeptions-why-they-spam-how-to-know-and-fix-it – Aaron Franke Nov 21 '16 at 23:00
  • I decided to try and report this bug to the Linux kernel devs, if someone could take a look and see if this is the correct place to report bugs and maybe add things to it then I would appreciate it :) https://bugzilla.kernel.org/show_bug.cgi?id=188331 – Aaron Franke Nov 21 '16 at 23:26

4 Answers4

7

There are two types of file access that use disk space, but don't show up with your tools: deleted (but still open) files, and files being written to.

I have these two aliases defined that I find very useful: 

# from http://www.certpal.com/blogs/2010/12/find-open-files-in-linux-using-lsof/
alias bigopenfiles="sudo lsof / | awk '{if(\$7 > 1048576) print \$7/1048576 \"MB\" \" \" \$9 }' | sort -n -u" 

alias deletedfiles="sudo lsof / | egrep 'PID|\(deleted\)'"
waltinator
  • 36,399
  • I think you've got something here! bigopenfiles returns 11222.7MB /var/log/kern.log 11222.9MB /var/log/syslog! However, sudo rm /var/log/syslog doesn't work! – Aaron Franke Nov 21 '16 at 20:09
  • Restart your system log program and see if the file goes away. – Mark Nov 21 '16 at 23:55
  • Read man -k logrotate - it will show you how to keep logs from filling up /. Next time, consider separate partitions for files that can fill up. – waltinator Nov 22 '16 at 00:13
  • 1
    rm can remove the name of a file from a directory, but if it's currently open by any process (whether for reading or writing), it'll stay on the disk until the process releases it. Some programs can be sent a signal to tell them to flush, close and re-open their log files; others must be restarted. – deltab Nov 22 '16 at 03:51
  • I want to add that I "fixed" this by mounting /var/log on a separate partition. It still constantly fills up :/ – Aaron Franke Nov 28 '16 at 21:16
  • Check what is filling your logfiles with cut "-d " -f5- /var/log/syslog /var/log/kern.log | sed -e 's/[0-9]/#/g' | sort | uniq -c | sort -rn – waltinator Nov 29 '16 at 21:17
  • It appears to be a kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=188331 and https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1643719 – Aaron Franke Mar 12 '17 at 22:08
  • @Mark What is my system log program? How can I find out? – Michael Levy Jan 11 '22 at 19:49
  • @waltinator. After I check what is filling my logfiles with [], what do I do then? – Michael Levy Jan 11 '22 at 19:51
2

Your /var is very fat. Check /var/log

You can use this command to find big files :

sudo find /var -xdev -type f -size +500000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

of course you can adjust the size (500000k in this sample)

After that, you can remove or compress them

f35
  • 945
  • Your command returned nothing. I've already tried sudo rm -R /var/log to no effect. I just tried again and it didn't work. I've updated my original post. – Aaron Franke Nov 21 '16 at 20:06
  • @f35, can I delete without problem the list I obtained from your find... ? Can I automate the process? – Michael Levy Jan 11 '22 at 19:54
  • No you can't ! This command returns the list of big files, you have to check file by file which one is useless – f35 Jan 13 '22 at 07:05
2

You're likely not seeing it with your tools because the file is open. Try this as root

> /var/log/syslog

Exactly, including the >. This will truncate the log.

NOTE:

  • This WILL erase the log!
Michael Sandman
  • 1,771
0

As @waltinator told here if you find your deleted syslog files occupying more space even after deleting them and if you can't free it with rm -R /var/log or > /var/log/syslog, kill the Syslog daemon, (in Elementary OS (Ubuntu 18.04), I did killall rsyslogd to release the open files and freed around 15Gb of / space)

It would be better to mount /var in a separate partition if you've one.

Vaibhav S
  • 1