0

So here are mt symptoms. Over the last few days I start the day with say 33 GB of space and then come in the next day to see that I have only 1GB of space left. I am assessing space by using "df"

hari@hari-Precision-WorkStation-T7500:~$ df
Filesystem                                                              1K-blocks       Used  Available Use% Mounted on
/dev/sde1                                                               230398116  216818952    1852572 100% /
none                                                                            4          0          4   0% /sys/fs/cgroup
udev                                                                      6135136          4    6135132   1% /dev
tmpfs                                                                     1229864       1588    1228276   1% /run
none                                                                         5120          0       5120   0% /run/lock
none                                                                      6149304       5808    6143496   1% /run/shm
none                                                                       102400         56     102344   1% /run/user
//server01.mycompany.com/archive$/Biology/RAW-Data 8458860528 4685045352 3773815176  56% /home/rawdata2
/dev/sda1                                                              1953512032 1724815580  228696452  89% /media/hari/FreeAgent GoFlex Drive1

The only change I did to my system was mount a 9 TB fileshare using the cifs (samba) module before this problem started.

When I run the disk usage analyser "baobab", it reports on the disk without suggesting that any single directory under my home has grown overnight significantly - i.e the percentages don't go up significantly.

I am running a private webserver on this machine and not running much else on this machine. There are no logs that are growing significantly overnight nor are there any processes that have much disk io as investigated by "htop" "top".

I tried to use "sudo iotop" but I dont know how to interpret its output.

Where is my disk space going. I am scared it is some indexer that is trying to index the 9TB fileshare that is holding on to some space since the problem started after I added the 9TB fileshare.

I am running Ubuntu 13.10 , 3.11.0-15-generic

How do I troubleshoot whats using all this space. I am scared its some indexer or some other background process.

harijay
  • 217
  • Out of curiousity, where are you mounting the cifs share? Under /home by chance? – Mordoc Jan 17 '14 at 17:33
  • Yes the cifs share is mounted under /home/rawdata2 – harijay Jan 17 '14 at 17:42
  • So out of curiosity, if you unmount the share does the space return? Either way another approach would be to mount in a location outside of /home say under /media. – Mordoc Jan 17 '14 at 18:01
  • I unmounted and removed all mount commands from /etc/fstab etc and then re-booted --the space does not return. I will change the mount points to /media. I now have 6.8GB free..hope this stays the same tommorrow. The previous few days the space remaining "disappeared" overnight. – harijay Jan 17 '14 at 18:16
  • 1
    It seems much more a kind of log-files spam. Try to run the disk analyzer on directory /var and /tmp. Also check your hidden directories and files --- cd; du -h -d2 . will give you the list of directories with their size (du -s * skips hidden ones). – Rmano Jan 17 '14 at 19:05
  • 1
    Sorry, du -h -d1 . (recurse 1 level under this one). – Rmano Jan 17 '14 at 19:33
  • The additional files are not due to an expansion of /var /tmp /home . Some other part of my filesystem is building up and taking up space. I am scared some indexer is working extra hard and storing its data somewhere in a manner that is hidden from du ( /home remains constant at 91GB ..but disk space df went from 13GB free to 1GB free again) – harijay Jan 23 '14 at 23:27
  • If you are in a situation where reboot isn't always freeing the spacing, I would reboot into rescue mode and use du as described above to get a feel for where the space is going. I tend to do "du -sc * | sort -n" as a quick-and-dirty way of doing this sort of investigation (dirty due to issues with hidden files, etc. as discussed above.) Start in the root and work down. – Steve Dee Jan 23 '14 at 23:54
  • 1
    I swear I wrote a question on this: http://askubuntu.com/questions/397842/how-to-find-the-biggest-directories-in-my-hard-drive – Kaz Wolfe Jan 24 '14 at 05:23
  • Last time I was running low on space my .x-session file was getting spammed to around 90 GB's over about a day. Someone was trying to "hack" into my remote desktop service, which all got logged. Maybe you are getting log-spammed like @Rmano said. – Dan Johansen Jan 24 '14 at 13:56

2 Answers2

3

If the disk space is always freed after reboot that most likely means that the space is used up by some unlinked files. Unlinked files are deleted = unlinked from the file system, but a process still keeps them open, so the disk space used up by such a file is not freed until the process releases the file. The process holding the unlinked file may also keep writing to the file and using up more and more disk space undetectable in the file system.

To view unlinked files and the process which holds them you can use lsof. Until a long time I always used some grep-ing on the lsof output to view the unlinked files, but a while ago I found this super answer on how to view unlinked files easily. You just have to use

lsof +L1

as this will list all open files which have link count less than 1, hence they are unlinked.

So use lsof +L1 and check if there are a lot of files or big files unlinked on your machine, you will also see which process holds them.

Community
  • 1
falconer
  • 15,026
  • 3
  • 48
  • 68
  • Thanks for your answer. When I go to /media I see a very strange behavior when I run "df -d1 -h " I get output: . 24K ./hari 4.0K ./cnstnewest 114G . So it seems that there are 114G of files in media without any external disks mounted. lsof +L1 gives very few files : all owned by chrome and gnome terminal. – harijay Jan 23 '14 at 23:49
  • 1
    @harijay And what does ls -lah /media show? – falconer Jan 23 '14 at 23:59
  • I think I am close to resolving this. I had created a permanent mount point in media for my external disk "FreeAgent GoFlex Drive" . I also had a crontab entry that was intended to backup to that external disk. At some point the external disk mount point changed to "FreeAgent GoFlex Drive1" and rsync created a mirror of my data on confusingly called /media/FreeAgent GoFlex Drive. I always failed to read the du output correctly It clearly said "du: cannot read directory ‘./hari/FreeAgent GoFlex Drive’: Permission denied". – harijay Jan 24 '14 at 15:42
  • @harijay Yes, that is likely your problem. I'm happy you solved it. :) – falconer Jan 24 '14 at 15:47
  • Add to that the Strange Ubuntu practice of automounting a drive at /media/hari/FreeAgent GoFlex Drive1 and calling the nautlius menu and the drive name without the 1 suffix. So I was all confused since I had a /media/FreeAgent GoFlex Drive and a ubuntu created /media/hari/FreeAgent GoFlex Drive1 which was called "FreeAgent GoFlex Drive". Also I always failed to read the first few lines of my du -h1 -h on /media which clearly said where 114GB was "missing" – harijay Jan 24 '14 at 17:51
  • 1
    @harijay Although I'm glad and appreciate that you accepted my answer, but since it wasn't really my answer which solved your problem I think it is better if you don't accept it. Better would be if you write an answer of your own, explaining the solution to your problem and accept that one. Or you can accept Steve Dee's answer below as it writes about the same technique you used to find out the problem, although it is not the full solution to your problem. You can still give the bounty to me if you really want, that is not tied to the accepted answer, just click on the award bounty icon. – falconer Jan 24 '14 at 18:09
  • @harijay But if you want it this way no problem, the OP decides. :) – falconer Jan 24 '14 at 18:10
  • I too thought about this a bit and did learn from the sort approach that Steve Dee answer below as with Rmanos comment above. If only I could split the bounty or accept multiple answers. People like you are rare... – harijay Jan 25 '14 at 23:04
  • @harijay Yes I know I'm a rare species. :) Decide just as you like. – falconer Jan 25 '14 at 23:15
1

If you are in a situation where reboot isn't always freeing the spacing, I would reboot into rescue mode and use du as described above to get a feel for where the space is going. I tend to do "du -sc * | sort -n" as a quick-and-dirty way of doing this sort of investigation (dirty due to issues with hidden files, etc. as discussed above.) Start in the root and work down.

I would agree this is a very odd problem. Perhaps try selectively disabling services to see if you can get it to stop? As far as indexing goes, the only things running system-wide as standard that I can think of should be locate/mlocate. There's also apt-xapian which indexes packages, mandb which indexes man pages.

Are you logging in to this machine with a desktop session? If so, does the problem stop if you do log in (at least graphically)?

Steve Dee
  • 837
  • 4
  • 9
  • Thanks Steve. I observed exactly what you described. Reboots were not freeing up any space. In some cases a tune2fs - m0 on the drive freed up some space. In the end I systematically went from directory to directory as you recommended and a combination of du -sc * | sort n and "du -d1 -h " helped me diagnose a directory in /media whose name corresponded to my external disk name "Freegent GoFlex Drive" which I mistook for my external disk. It had 144 GB of my "free" space because rsync created it due to a crontab job that ran everynight at 3.00 am. I also missed seeing it in my df runs – harijay Jan 25 '14 at 23:10