Ubuntu thinks Btrfs disk is full but it's not

Question

$ cat /etc/fstab
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=a168d1ac-4e13-4643-976d-6e47ea1732b1 /boot        ext2  defaults                                                                   0 1
/dev/mapper/sda4_crypt                    /            btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@          0 2
/dev/mapper/sda4_crypt                    /tmp         btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@tmp       0 2
/dev/mapper/sda4_crypt                    /run         btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@run       0 2
/dev/mapper/sda4_crypt                    /var/crash   btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@var-crash 0 2
/dev/mapper/sda4_crypt                    /var/tmp     btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@var-tmp   0 2
/dev/mapper/sda4_crypt                    /var/log     btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@var-log   0 2
/dev/mapper/sda4_crypt                    /var/spool   btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@var-spool 0 2
/dev/mapper/sda5_crypt                    /home        btrfs defaults,autodefrag,compress=lzo,inode_cache,space_cache,subvol=@home      0 3
/dev/mapper/750er                         /media/750er ext4  defaults                                                                   0 4
/dev/mapper/cswap                         none         swap  defaults                                                                   0 5
➜  ~  df -h         
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/sda4_crypt   38G   12G   13M 100% /
none                    4,0K     0  4,0K   0% /sys/fs/cgroup
udev                    2,0G  4,0K  2,0G   1% /dev
tmpfs                   396M  1,3M  394M   1% /run
none                    5,0M     0  5,0M   0% /run/lock
none                    2,0G  208K  2,0G   1% /run/shm
none                    100M   36K  100M   1% /run/user
/dev/mapper/sda4_crypt   38G   12G   13M 100% /tmp
/dev/sda2               231M   44M  175M  21% /boot
/dev/mapper/sda4_crypt   38G   12G   13M 100% /var/crash
/dev/mapper/sda4_crypt   38G   12G   13M 100% /var/tmp
/dev/mapper/sda4_crypt   38G   12G   13M 100% /var/log
/dev/mapper/sda4_crypt   38G   12G   13M 100% /var/spool
/dev/mapper/sda5_crypt  3,7T  2,4T  1,2T  67% /home
/dev/mapper/750er       688G  276G  377G  43% /media/750er
/dev/mapper/2tb         1,8T  1,7T  141G  93% /media/2tb
➜  ~  sudo btrfs fi df /
Data, single: total=9.47GiB, used=9.46GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=13.88GiB, used=1.13GiB
Metadata, single: total=8.00MiB, used=0.00
➜  ~  

It's a 40GB partition with quite some snapshots on it. But it's compressed so I think the 9,46GB/40GB is accurate. But my Ubuntu fails because it says it has no disk space. I had apt-errors, cant install programs and my mysql server failed to start because of it.

And I know about not to rely on df I just included it for completeness.

I think Ubuntu uses df which is known to report wrongly with Btrfs internally and fails for that reason. That would make sense for APT when it does checks for space. But it actually fails to write to disk.

$ sudo time dd if=/dev/zero of=large bs=2G count=1
dd: error writing ‘large’: No space left on device
0+1 records in
0+0 records out
11747328 bytes (12 MB) copied, 1,29706 s, 9,1 MB/s
Command exited with non-zero status 1
0.00user 1.40system 0:01.44elapsed 97%CPU (0avgtext+0avgdata 2098028maxresident)k
160inputs+23104outputs (0major+383008minor)pagefaults 0swaps

Answer 1

Btrfs is different from traditional filesystems. It is not just a layer that translates filenames into offsets on a block device, it is more of a layer that combines a traditional filesystem with LVM and RAID. And like LVM, it has the concept of allocating space on the underlying device, but not actually using it for files.

A traditional filesystem is divided into files and free space. It is easy to calculate how much space is used or free:

|--------files--------|                                                |
|------------------------drive partition-------------------------------|

Btrfs combines LVM, RAID and a filesystem. The drive is divided into subvolumes, each dynamically sized and replicated:

|--files--|    |--files--|         |files|         |                   |
|----@raid1----|------@raid1-------|-----@home-----|metadata|          |
|------------------------drive partition-------------------------------|

The diagram shows the partition being divided into two subvolumes and metadata. One of the subvolumes is duplicated (RAID1), so there are two copies of every file on the device. Now we not only have the concept of how much space is free at the filesystem layer, but also how much space is free at the block layer (drive partition) below it. Space is also taken up by metadata.

When considering free space in Btrfs, we have to clarify which free space we are talking about - the block layer, or the file layer? At the block layer, data is allocated in 1GB chunks, so the values are quite coarse, and might not bear any relation to the amount of space that the user can actually use. At the file layer, it is impossible to report the amount of free space because the amount of space depends on how it is used. In the above example, a file stored on the replicated subvolume @raid1 will take up twice as much space as the same file stored on the @home subvolume. Snapshots only store copies of files that have been subsequently modified. There is no longer a 1-1 mapping between a file as the user sees it, and a file as stored on the drive.

You can check the free space at the block layer with btrfs filesystem show / and the free space at the subvolume layer with btrfs filesystem df /

---

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/sda4_crypt   38G   12G   13M 100% /

For this mounted subvolume, df reports a drive of total size 38G, with 12G used, and 13M free. 100% of the available space has been used. Remember that the total size 38G is divided between different subvolumes and metadata - it is not exclusive to this subvolume.

# btrfs filesystem df /
Data, single: total=9.47GiB, used=9.46GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=13.88GiB, used=1.13GiB
Metadata, single: total=8.00MiB, used=0.00

Each line shows the total space and the used space for a different data type and replication type. The values shown are data stored rather than raw bytes on the drive, so if you're using RAID-1 or RAID-10 subvolumes, the amount of raw storage used is double the values you can see here.

The first column shows the type of item being stored (Data, System, Metadata). The second column shows whether a single copy of each item is stored (single), or whether two copies of each item are stored (DUP). Two copies are used for sensitive data, so there is a backup if one copy is corrupted. For DUP lines, the used value has to be doubled to get the amount of space used on the actual drive (because btrfs fs df reports data stored, not drive space used). The third and fourth columns show the total and used space. There is no free column, since the amount of "free space" is dependent on how it is used.

The thing that stands out about this drive is that you have 9.47GiB of space allocated for ordinary files of which you have used 9.46GiB - this is why you are getting No space left on device errors. You have 13.88GiB of space allocated for duplicated metadata, of which you have used 1.13GiB. Since this metadata is DUP duplicated, it means that 27.76GiB of space has been allocated on the actual drive, of which you have used 2.26GiB. Hence 25.5GiB of the drive is not being used, but at the same time is not available for files to be stored in. This is the "Btrfs huge metadata allocated" problem. To try and correct this, run btrfs balance start -m /. The -m parameter tells btrfs to only re-balance metadata.

A similar problem is running out of metadata space. If the output had shown that the metadata were actually full (used value close to total), then the solution would be to try and free up almost empty (<5% used) data blocks using the command btrfs balance start -dusage=5 /. These free blocks could then be reused to store metadata.

For more details see the Btrfs FAQs:

• I get "No space left on device" errors, but df says I've got lots of space

• Btrfs claims I'm out of space, but it looks like I should have lots left.

Answer 2

Short answer: Btrfs partition metadata is shown as 'used' by Standard disk utilities like df.

1. check the problem volume. for example: /

   btrfs subvolume list /
   

2. Most likely snapshots are filling the volume. Delete snapshot you do not need. keep one from last date you are sure system was working ok.

   btrfs subvolume delete <path> 
   

   Where path is from previous command subvolume list that says "snapshot".

3. Reboot and you are done

Reason for problem might be your distro or package manager making snaphots every time you update system.

NB: balance command fails if disk is full as there is no free space to blance to.

Answer 3

In my case the disk usage would not go down even when I deleted files and snapshots.

btrfs balance (data and metadata) did not work with error "no space left on device"

btrfs balance start -m /
ERROR: error during balancing '/': No space left on device
There may be more info in syslog - try dmesg | tail

The RAID1 showed full usage on both disks even though actual data usage was lower than a third of that.

# btrfs fi sh
Label: none  uuid: 61a20f1a-c133-11e6-964b-d3bac0c48bbd
    Total devices 2 FS bytes used 153.94GiB
    devid    1 size 455.76GiB used 455.76GiB path /dev/sda2
    devid    2 size 455.76GiB used 455.76GiB path /dev/sdb2


# btrfs filesystem df /
Data, RAID1: total=452.73GiB, used=151.51GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=3.00GiB, used=2.42GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Solution: discard empty chunks, needs no additional space:

btrfs balance start -dusage=0 /

btrfs balance start -musage=0 /

Source: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance#ENOSPC

Alternative: My solution was to shrink the disks See: https://unix.stackexchange.com/questions/239765/how-to-fix-btrfs-superblock-error-after-resize-shrink-btrfs-couldnt-get-super

btrfs filesystem resize 1:430g /
btrfs filesystem resize 2:430g /

(commands take time, check syslog to see relocating blocks)

after that resize up:

btrfs filesystem resize 1:450g /
btrfs filesystem resize 2:450g /

After that, btrfs balance (metadata) worked again:

btrfs balance -m /

Then btrfs balance of the data (relocate data-chunks that have lower than 33% usage):

btrfs balance -dusage=33 /

Answer 4

Credit goes to @ignis and @bain. Just to have a simple straight to the point reference answer here without all the lecture and to share what I actually did to get system working again.

btrfs balance start -m /mountpoint

Is the magic line to solve issues like this.

I ran into some trouble that I don't want to bore you with and I don't know it was necessary to run this from a live cd but what I did in the end after the system was messed up not booting was running btrfsck on the devices (unlocked crypto mappers) and it did actually found errors, then mounted the root btrfs filesystem without any options to /mnt, as opposed to on my installed system where / is the only the mounted @ subvolume. So I had all the snapshots and other subvolumes also in there. I do not know of this makes a difference.

btrfs balance start -m /mnt

I got the metadata from

Metadata, DUP: total=13.88GiB, used=1.13GiB

to

Metadata, DUP: total=1.38GiB, used=1.05GiB

And a refreshing:

$ sudo df -h /
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/sda4_crypt   38G   12G   26G  31% /

So I think everything is fine now.

Answer 5

In general, don't trust the command df when it comes to btrfs. Lets take an extremely simple example. Say have a btrfs on a 500 GiB partition. You create a subfolder foo which you add 300 GiB of data. You then create another subfolder of bar which you add 198 GiB. If overhead is neglishable, df would correctly show you have 2 GiB of space available.

Your grandmother just gave you 64 GiB of home videos to add. So you decide to delete the subfolder foo. Now chances are df still shows you have 2 GiB available. If you try to copy the video data with say SAMBA from a Windows computer, you will likely get an insufficient space error. But if you are logged into your local machine, you can successfully copy the 64 GiB without any errors.

What is going on her is at least last time I tried, SMBD (Samba) uses df to tell Windows how large the disk is, and how much space you have available. Windows does not want to copy the files if you don't have enough space, so it throws an error. But because df does not know how to inspect btrfs, it just reports the number of clusters that are listed as unallocated as the free space. But in reality, 300 GiB of clusters have been allocated, but are unused. So you really have 302 GiB of free disk space.

Now the solution is you can either configure samba to use the btrfs filesystem df command, or you regularly use btrfs balance start and play around until you find the correct options for your needs. Since I already configured samba to use my own command to report space available on submounts, I picked the first option, but the second option is easier.

Now unfortunately there are many applications such as apparently docker that will use df to throw errors similar to the way samba does. So most people go for regularly rebalancing their filesystem. And it does overall increase performance. But is is also time consuming when you want to copy the files NOW.