0

Ubuntu 14.04 on Amazon AWS EC2, has been in production for several years as a web server.

I was working on implementing a new backup process, and while doing so the server ran out of disk space. A 300gb magnetic drive, it's usage was at 290gb when I started this project. While backing up a rather large account, the drive ran out of space but was still mostly operational. OK, so I started deleting files. I deleted a lot of "temp" files local to a few particular web sites, I deleted some working backups (.gz files) that amounted to several gb, and I also checked the /tmp system folder and looked to see if there were any .Trashes folders. I believe in total I deleted at least 10gb of files. However my system still reported zero free space. And MySQL wouldn't start (which I know requires free space to start and operate).

I rebooted the server, to no avail. I shut down apache to make sure it wasn't holding open files. Eventually I expanded the disk drive to 350gb, and then everything started working. The server currently reports 298gb used. Should be much less than that, given all that I deleted.

1) Why would the server not recognize the space I was freeing up? 2) How can I get it to see it now?

Here is the output of df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            1.8G   12K  1.8G   1% /dev
tmpfs           358M  200K  358M   1% /run
/dev/xvda1      345G  283G   47G  86% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            1.8G     0  1.8G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M   76K  948K   8% /tmp
Rob Brandt
  • 225
  • And while it seemed to be working when I wrote the above, I just now see that I cannot create a cron job because of lack of space. – Rob Brandt May 15 '18 at 17:23
  • Can you edit your question to include the output of df -h please? – muclux May 15 '18 at 17:57
  • After changing the size of the physical disk, did you run resize2fs to resize the filesystem? Read man resize2fs. – waltinator May 16 '18 at 04:14
  • @waltinator, this is an issue that is confusing to me. I have been using Ubuntu on AWS since the 10.04 days, and until I installed a 16.04 server I have NEVER had to resize the file system. It doesn't appear that I need to now either. When I run lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL it shows just one entry, at 350gb. When I run fdisk -l, I get similar data, ending with "Disk /dev/xvda1 doesn't contain a valid partition table". So I guess since my disk isn't split between partitions, I don't have to resize(?).

    So I guess the reason I don't need to resize is that my disk i

     – Rob Brandt May 16 '18 at 21:02
  • I believe i have solved the "out of disk space" issue from here https://askubuntu.com/questions/199565/not-enough-space-on-tmp using option 2 from the approved answer. You can see from my table above that /tmp is mapped to overflow, which apparently is a stopgap Ubuntu does when it runs out of disk space. And then I ran out of overflow space. The solution was to clean out more disk space (down to 280gb used) and reboot. overflow is now gone and it appears /tmp is using regular disk space again. – Rob Brandt May 16 '18 at 21:06

0 Answers0