System drive (1TB) 82% full? All "my" files are on separate HDD

Question

This is a desktop system, with a 1TB SSD for the OS Xubuntu 20.04.6, and several HDDs for all my files. So the OS shouldn't be taking up that much space. But there's the output of df:

Filesystem      1K-blocks       Used  Available Use% Mounted on 
udev             16332704          0   16332704   0% /dev
tmpfs             3284264       3384    3280880   1% /run
/dev/nvme0n1p2  959786032  740380052  170577772  82% /

There's more lines, obviously, but the the /dev/nvme one is the problem. I can't for the life of me find what's taking up all that space. The "home" folder has about 6GB in it.

I'm suspicious of the "proc" directory. Nautilus says it has tens of thousands of files, for a total of 128TB. I think not. Here's the output of cd /proc; sudo du -hs

du: cannot read directory './3032/task/3032/net': Invalid argument
du: cannot read directory './3032/net': Invalid argument
du: cannot read directory './3288/task/3288/net': Invalid argument
du: cannot read directory './3288/net': Invalid argument
du: cannot access './6606/task/6606/fd/3': No such file or directory
du: cannot access './6606/task/6606/fdinfo/3': No such file or directory
du: cannot access './6606/fd/4': No such file or directory
du: cannot access './6606/fdinfo/4': No such file or directory
0

I used "du -hs" on all the other directories under "/" and they add up to about 55GB. And there's a 2GB swapfile. So how do I find what's eating up all my SSD space?

UPDATE: Here is the output of cd / ; sudo du -hxd 1

j@puter:/$ cd / ; sudo du -hxd 1
11M ./root
18M ./bin
20M ./sbin
4.0K    ./mnt
5.4G    ./usr
16K ./lost+found
184K    ./tmp
532M    ./opt
248M    ./boot
84K ./snap
12G ./var
6.4G    ./home
4.0K    ./lib64
8.0K    ./srv
1.5G    ./lib
4.0K    ./cdrom
14M ./etc
8.0K    ./media
28G .
j@puter:/$ 

Update: booted from a live USB, ran fsck

xubuntu@xubuntu:~$ sudo fsck -f /dev/nvme0n1p2
fsck from util-linux 2.37.2
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Inode 34099073 extent tree (at level 1) could be narrower.  Optimize<y>? yes
Inode 34344578 extent tree (at level 1) could be shorter.  Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p2: 514456/61022208 files (4.6% non-contiguous), 189197520/244059136 blocks
xubuntu@xubuntu:~$ 

Update: ncdu confirms what du tells me:

Total disk usage:  27.4 GiB  Apparent size:  27.2 GiB  Items: 402910

And when I look at "properties" in the GUI, it says

401685 items, totalling 26.1 GiB (28,044,956,192 bytes)
(some contents unreadable)
162.7 GiB of 915.3 GiB free (82% used)

Answer 1

Finally, found this on Ubuntuforums:

du - Estimate file space usage (Check your folders)

This command will show you how much space is being used by the mounted file systems on a given folder. One disadvantage of du is that it reports information only on mounted files. Additionally, if a device is mounted to a folder which already contains data, the underlying data size will not be included in the results. This means that if /media/data contains 40GB of data, but a 20GB partition is then mounted to /media/data, the results will show only the 20GB of the last-mounted device.

(emphasis added)

In my case, my internal data HDD (not the system SSD with the "problem") is backed up to an external portable drive. The external backup mounts to /mnt/backup/ (specified in fstab). I suspect the backup script must have run sometime when the backup drive wasn't mounted, so it wrote the contents of my data drive to the system disk! Apparently du -x ignores the mounted data, but does not include the underlying data either.

So the solution was simple.

sudo umount -a

delete the contents of /mnt/backup/

sudo mount -a

And now I have this:

403352 items, totalling 25.7 GiB (27,570,012,200 bytes)
(some contents unreadable)
841.5 GiB of 915.3 GiB free (8% used)

Thank you to all who replied and gave me pointers for things to search and learn about.

Answer 2

According to the information you provided, /dev/nvme0n1p2 is in fact 82% full.

Run the command cd / ; sudo du -hxd 1 To find out what is taking so much space.

The h option makes it in human-readable format.

The x option ignores all other mounted filesystems on /.

The d 1 option sets the directory depth to 1, showing only the first level directory making it easier to read.

Answer 3

It could be, that you have Huge Block Size for a Node (Sector on Storage). At your install of Linux, you can choose your favorite Filesystem, and the Size of the Blocks.
Therefore, you have to consider what do you want to do with your Linux-Computer:

1. did you want a File Server, then it could be better to have a small Block Size if you have multiple/huge number of Files, or
2. you have a Game-Desktop, then it could be better to have a bigger Block Size if you have minimal number of Files, with many Data-Information's.

E.g.: if you have small Files, that are only 512 Bytes in size, but you have chosen a Block size of 1024 Bytes at Filesystem Setup, then the Filesystem will store the 512 Bytes into the 1024 Block, and the Rest of the Block is empty, and reserved for only for the entire File. So you have a Gap of 512 unused Space. Then, the next file is 10 Bytes, than you have a Gap of 1014 Bytes. And so on, and so forth...
To more files you have, to more Blocks could be have Gap(s)...

So, you have to made a mind map, before you install Linux or other Operating System(s).

I can not give ultimate Tips or Hints, because it is your decision, and I would not, that the People says after all: "Me, I had told this and that..." I hope you can understand my Opinion.