Ubuntu 18.04 server partition becomes full randomly

Question

I'm experiencing the following problem:

The partition /dev/sda1 is getting used 90% or 100% randomly. It's normally on 47% and 50%, but I don't understand why it sometimes jumps up to 100%.

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           796M  692K  795M   1% /run
/dev/sda1        29G   14G   16G  48% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda15      105M  3.6M  101M   4% /boot/efi
tmpfs           796M     0  796M   0% /run/user/1000

This happens at least once per weekend. My solution is to reboot the OS from command line, by typing reboot, but that is not something healthy for the server.

What could it be the reason?

Answer 1

One thing that fits your description is:

Your /tmp directory getting used up by the applications running on your server and becoming full. The /tmp directory gets emptied upon reboot... hence the reduced partition size.

To check the usage of the /tmp directory please, run the following command in the terminal:

sudo du -hsc /tmp

To watch the usage of the /tmp directory in real time, please run the following command in the terminal:

sudo watch du -hsc /tmp

To exit real time, please press Ctrl + C.

---

To empty the /tmp directory without rebooting, please run the following command in the terminal:

sudo rm -rf /tmp/{*,.??*}

If you get an error like bash: /usr/bin/sudo: Argument list too long, this means your /tmp directory contains more files then rm can handle at once. To overcome this please run the following command instead in the terminal:

sudo find /tmp/{*,.??*} -exec rm -rf {} \;

If you get errors during this operation like No such file or directory., please ignore them as files are being added and deleted by system all the time in the /tmp directory.

---

If this solves your issue:

You can add a cronjob to run the empty command daily.

---

If this is not the case:

I recommend you investigate what is eating up your space by running the following command in the terminal:

sudo du -hca --time / | grep -E "^[0-9\.]*[G]"

This will list all the directories and files on your system with sizes more than 1G along with their modification dates.

To check for certain sizes:

For example to list only directories with 3G size, please use the following command:

sudo du -hca --time / | grep -E "^[3][0-9\.]*[G]"

You can change the number [3] to check for directories with say 5G size like so:

sudo du -hca --time / | grep -E "^[5][0-9\.]*[G]"

and so on.

Please, inspect the output with focus on recently modified ones and check if you see anything out of the norm.

You may need to continue monitoring and noting the size changes until you reach the moment where your partition is nearly full. You will then have a better understanding on what is causing this issue and will be able to make a decision.