0

I am seeing some very strange results when using du on some of my folders.

For example:

#cd /storage/main_folder

#du -h --max-depth=2 subfolder1 | grep subfolder2


1.8G    subfolder1/subfolder2/subfolder31
5.3G    subfolder1/subfolder2/subfolder32
352M    subfolder1/subfolder2/subfolder33
76K     subfolder1/subfolder2/subfolder34
1.8G    subfolder1/subfolder2/subfolder35
4.0M    subfolder1/subfolder2/subfolder36
301M    subfolder1/subfolder2/subfolder37
9.4G    subfolder1/subfolder2


du -h --max-depth=1 subfolder1/subfolder2


1.8G    subfolder1/subfolder2/subfolder31
5.3G    subfolder1/subfolder2/subfolder32
352M    subfolder1/subfolder2/subfolder33
96G     subfolder1/subfolder2/subfolder34
1.8G    subfolder1/subfolder2/subfolder35
4.0M    subfolder1/subfolder2/subfolder36
301M    subfolder1/subfolder2/subfolder37
106G    subfolder1/subfolder2

As far a I can tell, there are no links or special files in subfolder2. The disk is a local RAID volume, ext4, mounted with rw,noatime,nodiratime,stripe=64 .

What could be causing this, and what can I do to fix the problem, and get the correct results?

Thank you!

[ Edit, because I received a link to this question : I am aware of apparent size vs block size. But in this case, note that:

  1. subfolder34 shows up as 76K in one output, and 96G in the other. Too big a difference to be explained by apparent size...
  2. I am using the exact same du command, the only difference is the working directory when running the command. So we can exclude differences due to the way different utilities compute sizes ]
Bogd
  • 179
  • Hard links, most likely. – muru Jan 19 '21 at 09:49
  • How would hard links change the size of the folder? Note that in both examples, I am looking at the exact same folder: /storage/main_folder/subfolder1/subfolder2, yet getting widely different results. – Bogd Jan 19 '21 at 13:52
  • 1
    Sure you are, but you're looking at it differently. If I have two folders each with hard links to the same file, each folder's disk usage is the size of the file, but the combined usage of both is also just the size of the file, since the content only exists once. – muru Jan 19 '21 at 14:35
  • 1
    @muru - look at subfolder34. It shows up as 76K in the first command output, and 96G in the second one. WAY too big of a difference to be explained by apparent size.

    As for hard links - even if they were in use, why would du report such a huge difference for the exact same folder? (notice that it is the exact same subfolder34 in both outputs - the only difference is the working dir when running the command).

    Don't get me wrong - I truly appreciate your comments, and thank you for them! It's just that I cannot wrap my head around this, and this is why I keep asking add'l questions.

     – Bogd Jan 19 '21 at 20:49
  • Did you read the last part of gertvdijk's answer? – muru Jan 20 '21 at 01:30
  • [Edit - @muru, for some reason, I simply didn't see your previous reply mentioning how du handles hard links. This is what I was looking for - if you would post that as an answe, I would gladly accept it. Thank you once again for your help!!] – Bogd Jan 20 '21 at 10:21

1 Answers1

1

It looks like @muru hit the nail directly on the head from his first comment - it was just difficult for me to understand how this was happening because of hard links.

du is smart enough to only count hard-linked files once - so in my first example, if it found the files in subfolder34 anywhere else, it wouldn't count them again towards the folder size.

In the second command, since I am only looking at subfolder2, it does not see the other hard links - so it counts the size of the files.

If I run du with -l (telling it explicitly to count hardlinks multiple times), I get the expected results:

#cd /storage/main_folder

#du -h --max-depth=2 subfolder1 | grep subfolder2


1.8G    subfolder1/subfolder2/subfolder31
5.3G    subfolder1/subfolder2/subfolder32
352M    subfolder1/subfolder2/subfolder33
76K     subfolder1/subfolder2/subfolder34
1.8G    subfolder1/subfolder2/subfolder35
4.0M    subfolder1/subfolder2/subfolder36
301M    subfolder1/subfolder2/subfolder37
9.4G    subfolder1/subfolder2


#du -h --max-depth=2 -l subfolder1 | grep subfolder2


1.8G    subfolder1/subfolder2/subfolder31
5.3G    subfolder1/subfolder2/subfolder32
352M    subfolder1/subfolder2/subfolder33
96G     subfolder1/subfolder2/subfolder34
1.8G    subfolder1/subfolder2/subfolder35
4.0M    subfolder1/subfolder2/subfolder36
301M    subfolder1/subfolder2/subfolder37
106G    subfolder1/subfolder2

Thank you, @muru, for the comments, and for your patience with my many questions! :) And if you would post your comment as an answer, I will gladly accept it.

Bogd
  • 179