0

I ran into an issue where my server simply filled up this week.

➜  ~ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G   19G     0 100% /
devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G  102M  7.8G   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda3       1.8T   94G  1.6T   6% /home

Now it's unusable for the most part.

My hosting provider doesn't want to shrink /dev/sda3 and extend /dev/root/ for me unfortunately, and I'm a newbie at server administration and I know I'll mess up things by experimenting. Alternatively I'd also want to know what filled up the root filesystem (and I can't simply search in /) and see how to stop it.

How can I do that step-by-step without losing data? If needed, I can go into rescue mode. My hoster is Kimsufi.

Thanks!

Rod Smith
  • 44,284
  • 7
  • 63
  • 105
shavit
  • 215
  • 1
    Exactly because you are new to server admin, you should not attempt partition resizing, but instead focus on figuring out why all available space is being eaten. Usually, it is because of some runaway log file in /var. So start with sudo du -h --maxdepth 1 /var. This will give you a list of subfolders of /var. Drill down until you find the largest file or subfolder, then report back here so we can help you solve it. – Jos Jul 18 '17 at 17:07
  • Seems like /var/lib/mysql uses up a whole 4.5G! Also, the argument was --max-depth, not --maxdepth. – shavit Jul 18 '17 at 17:11
  • 1
    Apologies for that. Sounds like someone turned on the MySQL log and forgot to turn it off. – Jos Jul 18 '17 at 17:13
  • It's not in the /var/log directory but rather in /var/lib though. I just checked and it's filled with directories that look like my server's databases' names. Will cleaning up obsolete databases be the solution in my case? – shavit Jul 18 '17 at 17:18
  • 1
    Yes, your MySQL databases are in directories, one each. drop database xxxx should get rid of them. – Jos Jul 18 '17 at 17:22
  • Fortunately I have back-ups on the 1.8T space just in-case something lkike this will happen, thanks! – shavit Jul 18 '17 at 17:23
  • Additionally, you might define mySQL tablespaces to be somewhere beyond /home and create new databases there. – ridgy Jul 18 '17 at 21:04

1 Answers1

1

Concerning the discussion in comments, it sounds like you're running a server that relies on a large (or at least largeish) database in /var. On such a computer, it's important to either split off /var (or a subdirectory of it) into a big enough partition to hold the database or ensure that root (/) is big enough. In fact, many servers do better with a separate /var partition than with a separate /home partition -- although your df output shows that you've got 94GB on /home, so that doesn't seem to be the case for you. Nonetheless, if you're running a MySQL server, 20 GB for root (/) and no separate /var partition sounds inadequate, so your problem is understandable. You might want to consult a MySQL expert for advice on how much disk space your database might be expected to consume.

In any event, it is possible to do a "live" resize of some filesystems; but increasing the size is far better supported than decreasing a filesystem's size, and it appears you'll need to do one of each operation. If you can log in as root, you should be able to unmount /home and shrink it from your regular boot. Be aware that this is a non-standard configuration for Ubuntu, so you'll need to enable direct root logins -- and you may want to disable them afterwards. (See this question and its answers for more on this.) Depending on your partitions' layout, there may be other complications, too. See this question and its answers for some information on resizing while you're logged in. If your /home partition comes after your root (/) partition, you might want to consider shrinking /home, creating a new partition following it, moving the contents of /var (or whatever subdirectory of /var is chewing up so much disk space) to the new partition, and then mounting it at /var (or its subdirectory, as appropriate). Because so many programs work on /var, moving all of /var in this way would likely require working from an emergency disk, but you might be able to do this from the main installation for a subdirectory of /var.

That said, I agree with an earlier comment that, as a relative newbie, this might not be the best thing to attempt. If you must do it, I strongly recommend you set up a computer (or virtual machine) that matches your current configuration as closely as possible so that you can practice. It's better to wipe out a virtual machine (or two or three) than to wipe out your real server!

Another time-honored, but somewhat ugly, approach is to relocate one or more directories to an existing partition, and rely on a symbolic link rather than mounting a new partition. You'd do something like this:

  1. Locate a directory on the root (/) partition that's chewing up a lot of space. For MySQL, I'd expect it to be /var/lib/mysql.
  2. Create a target directory on the /home partition -- say, /home/var/lib/mysql. Match its ownership and permissions to the original /var/lib/mysql directory.
  3. Shut down any software that's using /var/lib/mysql. I think it would be sudo service mysql stop to do this, but I'm not 100% sure of that.
  4. Type sudo mv /var/lib/mysql/* /home/var/lib/mysql/. This will move the files from /var/lib/mysql to the /home partition.
  5. Type sudo rmdir /var/lib/mysql to delete that directory. (If it won't delete, then chances are some files didn't get moved. Maybe they're dot files. You must fix this problem.)
  6. Type sudo ln -s /home/var/lib/mysql /var/lib/mysql. This creates a symbolic link so that programs that use /var/lib/mysql will still be able to access the files in the new location.
  7. Restart MySQL. sudo service mysql start should do this.

Caveat: I'm not familiar enough with MySQL to be 100% confident that this procedure will work properly for it. The use of symbolic links might give it fits. Also, I haven't tested this procedure as I've typed it, so I may have made a mistake in one (or more) of the commands.

As I'm not a MySQL expert, there may be some better way to do this for MySQL specifically (assume that's really the program that's causing problems.)

Rod Smith
  • 44,284
  • 7
  • 63
  • 105
  • 1
    A bit late, but after testing (and following your exact instructions, not a single change to command) everything worked perfectly - just got to this after my databases started lacking disk space again. Might want to remove the caveat because I confirmed it works :) – shavit Aug 31 '17 at 12:11