I have created a new Ubuntu 20.04 VM for my docker host. I then moved my backup over to the new host and restarted all my containers. Apart from docker, docker-compose and ncdu nothing else has been installed on this host.
The VM has a 450GB disk and apparently 340GB are in use. I can not locate what is using all this diskspace though. All docker relevant mounts are in /home/docker/services/. NCDU reports 114.6 GiB disk usage which does not agree with df -h.
So my question is, what is happening here? What is causing this discrepancy?
docker@docker:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 4.0M 13G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 440G 340G 78G 82% /
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sda2 976M 104M 806M 12% /boot
/dev/sdc1 157G 61M 149G 1% /elastic
/dev/loop0 55M 55M 0 100% /snap/core18/1880
/dev/sdb1 1.5T 81G 1.4T 6% /downloads
/dev/loop1 56M 56M 0 100% /snap/core18/1944
/dev/loop2 72M 72M 0 100% /snap/lxd/16099
/dev/loop3 32M 32M 0 100% /snap/snapd/10492
/dev/loop4 30M 30M 0 100% /snap/snapd/8542
/dev/loop5 68M 68M 0 100% /snap/lxd/18150
//192.168.178.200/4k_series 79T 66T 13T 85% /4k_series
//192.168.178.200/4k_movies 79T 66T 13T 85% /4k_movies
//192.168.178.200/Docker_Backup 79T 66T 13T 85% /docker_backup
//192.168.178.200/Coding 79T 66T 13T 85% /coding
//192.168.178.200/Music 79T 66T 13T 85% /music
//192.168.178.200/eBooks 79T 66T 13T 85% /ebooks
//192.168.178.200/Cloud 79T 66T 13T 85% /cloud
//192.168.178.200/Movies 79T 66T 13T 84% /movies
//192.168.178.200/Audiobooks 79T 66T 13T 85% /audiobooks
//192.168.178.200/Fred 79T 66T 13T 85% /fred
//192.168.178.200/Series 79T 66T 13T 85% /series
//192.168.178.200/Photos 79T 66T 13T 85% /photos
tmpfs 13G 0 13G 0% /run/user/1000
ncdu has been run as root output:
ncdu 1.14.1 ~ Use the arrow keys to navigate, press ? for help
--- / ------------------------------------------------------------------------------------------------------------------------------------------------------------
77.6 GiB [##########] /home
25.9 GiB [### ] /var
8.0 GiB [# ] swap.img
2.1 GiB [ ] /usr
1.0 GiB [ ] /snap
101.0 MiB [ ] /boot
5.3 MiB [ ] /etc
3.9 MiB [ ] /run
60.0 KiB [ ] /tmp
52.0 KiB [ ] /root
20.0 KiB [ ] /elastic
16.0 KiB [ ] /opt
e 16.0 KiB [ ] /lost+found
e 4.0 KiB [ ] /srv
e 4.0 KiB [ ] /mnt
e 4.0 KiB [ ] /media
e 4.0 KiB [ ] /incomplete
e 4.0 KiB [ ] /cdrom
. 0.0 B [ ] /proc
0.0 B [ ] /sys
0.0 B [ ] /dev
@ 0.0 B [ ] libx32
@ 0.0 B [ ] lib64
@ 0.0 B [ ] lib32
@ 0.0 B [ ] sbin
@ 0.0 B [ ] lib
@ 0.0 B [ ] bin
< 0.0 B [ ] series
< 0.0 B [ ] photos
< 0.0 B [ ] music
< 0.0 B [ ] movies
< 0.0 B [ ] fred
< 0.0 B [ ] ebooks
< 0.0 B [ ] downloads
< 0.0 B [ ] docker_backup
< 0.0 B [ ] coding
< 0.0 B [ ] cloud
< 0.0 B [ ] audiobooks
< 0.0 B [ ] 4k_series
< 0.0 B [ ] 4k_movies
Total disk usage: 114.6 GiB Apparent size: 128.1 TiB Items: 4296902
My fstab:
docker@docker:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-2qMn7z44yFIFNUpjG0awRAYnY86pTJbxy0DB3uCLpkalGZFRG023GYBeeazYfbe4 / ext4 defaults 0 0
Server mounts
//192.168.178.200/Fred /fred cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Audiobooks /audiobooks cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Series /series cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Movies /movies cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/eBooks /ebooks cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Music /music cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Coding /coding cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Docker_Backup /docker_backup cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Cloud /cloud cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Photos /photos cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/4k_series /4k_series cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/4k_movies /4k_movies cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
/boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/d47fc774-f9fe-40e8-a185-3a764ce6f971 /boot ext4 defaults 0 0
/swap.img none swap sw 0 0
secondary mounts
/dev/sdb1 /downloads ext4 defaults 0 0
/dev/sdc1 /elastic ext4 defaults 0 0
Output of du -hs /
sudo du -hs / --exclude /movies --exclude /series --exclude /photos --exclude /4k_series --exclude /4k_movies --exclude /fred --exclude /music --exclude /docker_backup --exclude /coding --exclude /cloud --exclude /audiobooks --exclude /ebooks --exclude /downloads --exclude /elastic
[sudo] password for docker:
du: cannot access '/proc/3471/task/355748/fd/12': No such file or directory
du: cannot access '/proc/5906/task/1546871/fd/449': No such file or directory
du: cannot access '/proc/1661328': No such file or directory
du: cannot access '/proc/1671145': No such file or directory
du: cannot access '/proc/1675436/task/1675436/fd/4': No such file or directory
du: cannot access '/proc/1675436/task/1675436/fdinfo/4': No such file or directory
du: cannot access '/proc/1675436/fd/3': No such file or directory
du: cannot access '/proc/1675436/fdinfo/3': No such file or directory
du: cannot access '/proc/1675471': No such file or directory
du: cannot access '/proc/1675472': No such file or directory
du: cannot access '/proc/1675480': No such file or directory
du: cannot access '/proc/1675510': No such file or directory
du: cannot access '/proc/1675511': No such file or directory
du: cannot access '/proc/1675513': No such file or directory
du: cannot access '/proc/1675514': No such file or directory
du: cannot access '/proc/1675515': No such file or directory
du: cannot access '/proc/1675516': No such file or directory
du: cannot access '/proc/1675522': No such file or directory
du: cannot access '/proc/1675523': No such file or directory
du: cannot access '/proc/1675637': No such file or directory
du: cannot access '/proc/1675640': No such file or directory
du: cannot access '/proc/1675641': No such file or directory
116G /
Unmounted all shares, no data in these directories:
docker@docker:~$ ls -lah /
total 8.1G
drwxr-xr-x 34 root root 4.0K Jan 1 12:28 .
drwxr-xr-x 34 root root 4.0K Jan 1 12:28 ..
drwxr-xr-x 6 docker docker 4.0K Dec 30 11:14 4k_movies
drwxr-xr-x 2 docker docker 4.0K Dec 29 14:24 4k_series
drwxr-xr-x 2 root root 4.0K Dec 27 13:46 audiobooks
lrwxrwxrwx 1 root root 7 Jul 31 16:28 bin -> usr/bin
drwxr-xr-x 2 root root 4.0K Dec 27 12:36 boot
drwxr-xr-x 2 root root 4.0K Dec 27 12:36 cdrom
drwxr-xr-x 2 root root 4.0K Dec 27 13:48 cloud
drwxr-xr-x 2 docker docker 4.0K Dec 27 13:47 coding
drwxr-xr-x 19 root root 4.2K Dec 31 14:01 dev
drwxr-xr-x 2 root root 4.0K Dec 27 13:48 docker_backup
drwxr-xr-x 2 root root 4.0K Dec 27 20:10 downloads
drwxr-xr-x 2 docker docker 4.0K Dec 27 13:53 ebooks
drwxr-xr-x 2 root root 4.0K Dec 27 20:11 elastic
drwxr-xr-x 100 root root 4.0K Jan 1 13:32 etc
drwxr-xr-x 3 root root 4.0K Dec 27 13:28 home
drwxr-xr-x 2 docker docker 4.0K Dec 27 13:42 fred
lrwxrwxrwx 1 root root 7 Jul 31 16:28 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Jul 31 16:28 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 9 Jul 31 16:28 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 Jul 31 16:28 libx32 -> usr/libx32
drwx------ 2 root root 16K Dec 27 12:36 lost+found
drwxr-xr-x 2 root root 4.0K Jul 31 16:28 media
drwxr-xr-x 2 root root 4.0K Jul 31 16:28 mnt
drwxr-xr-x 17 docker docker 4.0K Dec 30 10:01 movies
drwxr-xr-x 2 docker docker 4.0K Dec 27 13:42 music
drwxr-xr-x 3 root root 4.0K Dec 27 13:33 opt
drwxr-xr-x 2 docker docker 4.0K Dec 27 13:48 photos
dr-xr-xr-x 457 root root 0 Dec 31 14:01 proc
drwx------ 5 root root 4.0K Dec 29 22:35 root
drwxr-xr-x 31 root root 980 Jan 2 14:04 run
lrwxrwxrwx 1 root root 8 Jul 31 16:28 sbin -> usr/sbin
drwxr-xr-x 6 docker docker 4.0K Dec 30 03:52 series
drwxr-xr-x 6 root root 4.0K Dec 27 13:28 snap
drwxr-xr-x 2 root root 4.0K Jul 31 16:28 srv
-rw------- 1 root root 8.0G Dec 27 12:37 swap.img
dr-xr-xr-x 13 root root 0 Dec 31 14:01 sys
drwxrwxrwt 13 root root 4.0K Jan 2 14:04 tmp
drwxr-xr-x 14 root root 4.0K Jul 31 16:29 usr
drwxr-xr-x 13 root root 4.0K Jul 31 16:30 var
docker@docker:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 1.5M 13G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 440G 342G 77G 82% /
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/loop3 32M 32M 0 100% /snap/snapd/10492
tmpfs 13G 0 13G 0% /run/user/1000
du -hca --time output:
docker@docker:~$ sudo du -hca --time / --exclude /movies --exclude /series --exclude /photos --exclude /4k_series --exclude /4k_movies --exclude /joshua --exclude /music --exclude /docker_backup --exclude /coding --exclude /cloud --exclude /audiobooks --exclude /ebooks --exclude /downloads --exclude /elastic | grep -E "^[0-9\.]*[G]"
1.2G 2020-12-30 08:59 /usr/lib
2.1G 2020-12-31 13:33 /usr
3.4G 2021-01-04 08:53 /var/log/journal/a5e2984a17e543058168dfd78be80d72
3.4G 2021-01-04 08:53 /var/log/journal
3.5G 2021-01-04 08:53 /var/log
1.4G 2021-01-02 19:54 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/usr
1.5G 2021-01-02 19:58 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt/conda/lib
2.1G 2021-01-02 19:59 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt/conda
2.6G 2021-01-02 19:59 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt
4.0G 2021-01-03 04:02 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged
4.0G 2021-01-03 04:02 /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962
31G 2021-01-04 08:53 /var/lib/docker/overlay2
31G 2021-01-04 08:54 /var/lib/docker
32G 2021-01-04 08:54 /var/lib
35G 2021-01-04 08:54 /var
du: cannot access '/proc/13601/task/13907/net/dev_snmp6/vethbaf5745': No such file or directory
du: cannot access '/proc/1361014': No such file or directory
du: cannot access '/proc/1395411': No such file or directory
du: cannot access '/proc/1409809/task/1409809/fd/4': No such file or directory
du: cannot access '/proc/1409809/task/1409809/fdinfo/4': No such file or directory
du: cannot access '/proc/1409809/fd/3': No such file or directory
du: cannot access '/proc/1409809/fdinfo/3': No such file or directory
8.1G 2020-12-27 12:37 /swap.img
1.9G 2021-01-04 08:54 /home/docker/services/traefik/log/access.log
1.9G 2021-01-04 08:54 /home/docker/services/traefik/log
1.9G 2021-01-04 08:54 /home/docker/services/traefik
5.6G 2021-01-04 08:18 /home/docker/services/jellyfin/data/metadata/library
3.7G 2021-01-04 08:18 /home/docker/services/jellyfin/data/metadata/People
9.4G 2021-01-04 08:18 /home/docker/services/jellyfin/data/metadata
9.8G 2021-01-04 08:20 /home/docker/services/jellyfin/data
11G 2021-01-04 08:20 /home/docker/services/jellyfin
1.1G 2020-12-27 09:31 /home/docker/services/nextcloud/data
1.4G 2020-12-27 09:31 /home/docker/services/nextcloud
1.6G 2021-01-04 08:10 /home/docker/services/tautulli/backups
2.0G 2021-01-04 08:53 /home/docker/services/tautulli
8.8G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/TV Shows
9.3G 2021-01-03 01:14 /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/Movies
2.1G 2020-12-27 20:27 /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/Albums
21G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata
1.3G 2020-12-27 18:50 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-22
1.3G 2021-01-03 01:01 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2021-01-03
1.3G 2021-01-03 04:03 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db
1.3G 2020-12-27 18:50 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-25
1.3G 2020-12-28 01:01 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-28
8.2G 2021-01-04 08:55 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases
8.5G 2021-01-04 08:55 /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support
15G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder
15G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Cache
11G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Media/localhost
11G 2021-01-04 08:18 /home/docker/services/plex/Library/Application Support/Plex Media Server/Media
55G 2021-01-04 08:55 /home/docker/services/plex/Library/Application Support/Plex Media Server
55G 2021-01-04 08:55 /home/docker/services/plex/Library/Application Support
55G 2021-01-04 08:55 /home/docker/services/plex/Library
55G 2021-01-04 08:55 /home/docker/services/plex
3.2G 2021-01-04 06:32 /home/docker/services/influxdb/data/telegraf/autogen
3.3G 2021-01-04 07:18 /home/docker/services/influxdb/data/telegraf
3.9G 2021-01-04 08:51 /home/docker/services/influxdb/data
3.9G 2021-01-04 08:55 /home/docker/services/influxdb
4.0G 2021-01-03 15:12 /home/docker/services/radarr/MediaCover
4.4G 2021-01-04 08:55 /home/docker/services/radarr
81G 2021-01-04 08:55 /home/docker/services
81G 2021-01-04 08:55 /home/docker
81G 2021-01-04 08:55 /home
127G 2021-01-04 08:55 /
127G 2021-01-04 08:55 total
sudo du -hs /then go in to each directory and try again until you find the culprit:cd large_directory; sudo du -hs ./– moo Jan 01 '21 at 12:37ncduwithrootpermissions (sudo)? – Murphy Jan 01 '21 at 12:47ncdushows your/homefolder as being the most used. You can runncdu /home/$USERorncdu ~to show you which of your folders is the largest in there. – Terrance Jan 01 '21 at 15:57sudo du -hca --time / | grep -E "^[0-9\.]*[G]".More info here to help you detect what is consuming space the most. – Raffa Jan 04 '21 at 08:49/somewhere else (e.g., on/mnt:sudo mount -o bind / /mnt), and checkduoutput on that (sudo du -hd1 /mnt). It's possible that some huge directory is mounted over by something else and not accessible todudirectly. – muru Jan 04 '21 at 09:27du, or are you talking about some other directory? – muru Jan 04 '21 at 09:48