Amazon EC2 instance Ubuntu 16.04 : Cannot create temp file for here document. No space left on device

Question

I have an EC2 instance running Ubuntu 16.04, which keeps on running out of memory killing my mongod process in turn.

Doing df -h gives

udev            2.0G     0  2.0G   0% /dev
tmpfs           396M   41M  355M  11% /run
/dev/xvda1      7.8G  7.4G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/1000

so as my /dev/xvda1 is full. I go in my '/' or root directory and do a

ls -l --block-size=M 
total 1M
drwxr-xr-x   2 root root 1M Nov 22 04:49 bin
drwxr-xr-x   3 root root 1M Dec 21 13:14 boot
drwxrw-rwx   3 root root 1M Oct 18 21:01 data
drwxr-xr-x  16 root root 1M Oct 18 20:30 dev
drwxr-xr-x  91 root root 1M Dec 14 01:29 etc
drwxr-xr-x   3 root root 1M Oct 18 20:30 home
lrwxrwxrwx   1 root root 1M Dec 21 13:14 initrd.img -> boot/initrd.img-4.4.0-57-generic
lrwxrwxrwx   1 root root 1M Dec  6 05:03 initrd.img.old -> boot/initrd.img-4.4.0-53-generic
drwxr-xr-x  21 root root 1M Sep  7 19:24 lib
drwxr-xr-x   2 root root 1M Sep  7 19:22 lib64
drwx------   2 root root 1M Sep  7 19:26 lost+found
drwxr-xr-x   2 root root 1M Sep  7 19:22 media
drwxr-xr-x   2 root root 1M Sep  7 19:22 mnt
drwxr-xr-x   2 root root 1M Sep  7 19:22 opt
dr-xr-xr-x 139 root root 0M Oct 18 20:29 proc
drwx------   4 root root 1M Oct 18 21:00 root
drwxr-xr-x  23 root root 1M Dec 25 13:55 run
drwxr-xr-x   2 root root 1M Oct 19 06:11 sbin
drwxr-xr-x   2 root root 1M Sep  1 17:37 snap
drwxr-xr-x   2 root root 1M Sep  7 19:22 srv
dr-xr-xr-x  13 root root 0M Dec 25 13:59 sys
drwxrwxrwt  11 root root 1M Dec 25 14:17 tmp
drwxr-xr-x  10 root root 1M Sep  7 19:22 usr
drwxr-xr-x  14 root root 1M Oct 18 20:52 var
lrwxrwxrwx   1 root root 1M Dec 21 13:14 vmlinuz -> boot/vmlinuz-4.4.0-57-generic
lrwxrwxrwx   1 root root 1M Dec  6 05:03 vmlinuz.old -> boot/vmlinuz-4.4.0-53-generic

If I add up all the file size it doesn't add up to be 7.4 GB. Then what is ? and how do I fix this? So that it doesn't overflow and kill my mongod process in return.

On some of the answer. It said restart your system. Post restarting this is the output.

udev            2.0G     0  2.0G   0% /dev
tmpfs           396M  5.6M  390M   2% /run
/dev/xvda1      7.8G  5.3G  2.2G  72% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/1000

I still use 5.3G of space. what is causing 2 GB to be filled? How do I drill down the issue?

Latest Edit After starting 2 NodeJS process and a mongod process post restart. In less than 14 hrs it again off memory.

NOT A DUPLICATE
They were running a local instance of Ubuntu and I am not. The answer which helps drill down the cause, will not run here. It was stopped with error (gksudo:3647): Gtk-WARNING **: cannot open display:. Most of the answers here require somekind of GUI based tools. Which is a limitation here. Hence this is not a duplicate and I request it to be kept open.

EDIT 1

ubuntu@ip-172-31-29-235:/$ sudo du -hcsx .[!.]* * | sort -rh | head
du: cannot access '.[!.]*': No such file or directory
du: cannot access 'proc/4433/task/4433/fd/4': No such file or directory
du: cannot access 'proc/4433/task/4433/fdinfo/4': No such file or directory
du: cannot access 'proc/4433/fd/4': No such file or directory
du: cannot access 'proc/4433/fdinfo/4': No such file or directory
3.9G    total
1.8G    usr
983M    home
363M    lib
323M    data
249M    var
163M    boot
16M bin
13M sbin
5.8M    etc

If 3.9 GB is totally utilized.

ubuntu@ip-172-31-29-235:/$ df -h                                   
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           396M  5.6M  390M   2% /run
/dev/xvda1      7.8G  6.5G  936M  88% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/1000

and Used is 6.5 GB where did 2.6 GB go?

EDIT 2
Adding output of sudo lsof | grep deleted

ubuntu@ip-172-31-29-235:/$ sudo lsof | grep deleted
mongod    1272                ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1273           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1275           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1276           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1277           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1278           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1279           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1280           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1281           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1282           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1283           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1284           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1285           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1286           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1287           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1288           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1289           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1291           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1292           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1293           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1294           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1297           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1298           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1299           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1300           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1301           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1302           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1303           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1304           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1305           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1306           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1307           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1308           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1309           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1310           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1311           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1312           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1313           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1314           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1315           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1316           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1317           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1318           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1319           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1320           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1321           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1322           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1323           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1324           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1325           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1326           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1327           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1328           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1329           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1330           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1331           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1332           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1333           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1334           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1335           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1336           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1337           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1338           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1342           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 1753           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 2499           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)
mongod    1272 2500           ubuntu    4w      REG              202,1 2727852655     160748 /var/log/mongodb/mongod.log (deleted)

Could there be a running process holding a reference to a large unlinked file – like a database instance or a large log file with many repetitive messages? Those won't show up in du but they still occupy space until the process closes the file descriptor (process termination, maybe forceful, will achieve that too). — David Foerster, Dec 26 '16 at 10:44
@SarasArya also add the output of sudo lsof | grep deleted — muru, Dec 26 '16 at 10:46
@DavidFoerster how to do I get to the point of identifying such processes/files so I can close them down? — Saras Arya, Dec 26 '16 at 12:22
SarasArya there's the files @David hinted at. All those deleted files are still taking up space because mongod hasn't let go of them. — muru, Dec 26 '16 at 15:13
How do I permanently delete them and make sure that they don't reappear? Probably that could be written as an answer. Will appreciate it — Saras Arya, Dec 26 '16 at 19:43
@DavidFoerster How do I access these deleted files ? and how do I delete them permanently? — Saras Arya, Dec 28 '16 at 18:45

score 1 · Answer 1 · answered Dec 28 '16 at 20:17

According to the output of lsof there is a large (2.7 GB), unlinked (“deleted”) file that was once opened through the path /var/log/mongodb/mongod.log.

The only way to trigger the deletion of a file is to remove all references to it. That includes

unlinking all its directory entries (which you did) and
closing all file descriptors referring to it.

Only the process holding the file descriptor can close it; alternatively the kernel closes all remaining file descriptors when a process terminates or is terminated. Since the file in question is is the log file of and its descriptor held by the mongod service the best way to achieve that is to terminate the service (e. g. with sudo service mongod stop or, if the daemon process is unresponsive, with sudo killall -s KILL mongod).

This will not prevent the reoccurrence of the issue if the log file keeps getting filled. You need to look at the log content and investigate the issue of which mongod is trying to notify. Please open a new question if you need help with that.

Amazon EC2 instance Ubuntu 16.04 : Cannot create temp file for here document. No space left on device

1 Answers1