January 15, 20251 yr I have 3 drives in a cache pool. They are 512GB, 512GB, and 240GB. I would have expected that I would have just under half of the total of this as usuable cache space -- ie ~630GB. However, I end up with a lot less. # btrfs fi usage -T /mnt/cache Overall: Device size: 1.15TiB Device allocated: 425.09GiB Device unallocated: 752.36GiB Device missing: 0.00B Device slack: 0.00B Used: 423.11GiB Free (estimated): 376.27GiB (min: 250.88GiB) Free (statfs, df): 265.00GiB Data ratio: 2.00 Metadata ratio: 3.00 Global reserve: 264.48MiB (used: 0.00B) Multiple profiles: no Data Metadata System Id Path RAID1 RAID1C3 RAID1C3 Unallocated Total Slack -- --------- --------- --------- -------- ----------- --------- ----- 1 /dev/sdf1 210.00GiB 1.00GiB 32.00MiB 265.91GiB 476.94GiB - 2 /dev/sdj1 211.00GiB 1.00GiB 32.00MiB 264.91GiB 476.94GiB - 3 /dev/sdl1 1.00GiB 1.00GiB 32.00MiB 221.54GiB 223.57GiB - -- --------- --------- --------- -------- ----------- --------- ----- Total 211.00GiB 1.00GiB 32.00MiB 752.36GiB 1.15TiB 0.00B Used 210.90GiB 444.69MiB 48.00KiB # df /mnt/cache Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdf1 617322596 222101324 277876812 45% /mnt/cache # df /mnt/cache -h Filesystem Size Used Avail Use% Mounted on /dev/sdf1 589G 212G 266G 45% /mnt/cache The 589G from df corresponds to 617322596*1024 bytes, and that is exactly 50% of my total SSD capacity. My question is why am I losing close to 20% of that? The Used+Avail space is only 478GB, or 501219328*1024 bytes. The difference is 111GB... Even with the overhead of the filesystem, that's more of a haircut than I expected. I would think that all of my space is usable within btrfs raid1. It should use half of the 240GB (ie 120GB eaach) with each of the 512GB Drives, and then the remaining 389GB on those two drives could be paired to use all of the other space. What am I missing?
January 15, 20251 yr I was curious myself about this. I've never much paid attention to the overhead of a filesystem. So I talked it over with my AI and it comes down to this: The 1GB mentioned is the allocated space for metadata. However, due to how BTRFS RAID1 works, it generates and stores multiple copies of metadata across drives, which can lead to higher actual usage. This includes not just primary metadata but also snapshots, checksums, and various system overheads needed for data integrity. With the metadata ratio set to 3, each piece of metadata is stored three times across your drives. So with each drive storing its own metadata along with the metadata of the other 2 drives, it adds up.
January 15, 20251 yr Author I did configure the pool's metadata to Raid1c3, so there is 3 copies of metadata instead of the usual 2 that would correspond to raid1. The table above confirms this -- it shows 1G of metadata on each drive. However, that's still only a total of 3G. Even with 3G metadata and another 1% for other overhead, I still have over 90G 'missing'. The carfax btrfs calculator shows that I shouldn't have any unusable space -- https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=0&d=240&d=512&d=512 Maybe as the drive fills up the 'missing' data will shrink. I might do a test for that.
January 15, 20251 yr Author i figured it out -- It doesn't work like I thought. I originally thought it should all 'fit' like this: Drive 1 (512GB): Mirrors 120GB to Drive 3 (240GB) + 392GB to Drive 2 (512GB) = 512GB used. Drive 2 (512GB): Mirrors 120GB to Drive 3 (240GB) + 392GB to Drive 1 (512GB) = 512GB used. Drive 3 (240GB): Mirrors 240GB to each of the 512GB drives = 240GB used. --> Total usable space (excluding overhead): 512+240/2 = 632GB It turns out that btrfs does not support that kind of 'creative' block distribution. It can't 'plan' to have the 240GB split across two drives. For some fun, I tried asking ChatGPT. I'm pretty sure it lied to me. It said I would only have 480GB of usable space and said that this is how my data would be allocated: Drive 1 (512GB): Mirrors 240GB to Drive 3 (240GB) + 240GB to Drive 2 (512GB) = 480GB used. Drive 2 (512GB): Mirrors 240GB to Drive 3 (240GB) + 240GB to Drive 1 (512GB) = 480GB used. Drive 3 (240GB): Mirrors 240GB to each of the 512GB drives = 240GB used. --> Total usable space (according to chatgpt): 480GB The problem with the above is that it suggests that each of drive 1 and drive 2 are mirroring 240GB to drive 3... but drive 3 can't store 480GB ! I think that the real 'answer' that matches my data above (ie 512GB of usable storage) is this: Drive 1 (512GB): Mirrors 240GB to Drive 3 (240GB) + 272GB to Drive 2 (512GB) = 512GB used (0 unused) Drive 2 (512GB): Mirrors 272GB to Drive 1 (512GB) = 272GB used (and 240GB unused) Drive 3 (240GB): Mirrors 240GB to Drive 1 (512GB) = 240GB used (0 unused) --> Total usable space (excluding overhead): 512GB What I've learned is that my choice of drives is poor. The 240GB drive can be removed entirely from this pool, with the only impact being that the metadata would have to be backed up only once instead of twice. I would then have 512GB of mirrored data, which after overhead should still be over 500GB of accessible storage! Drive 1 (512GB): Mirrors 512GB to Drive 2 (512GB) = 512GB used (0 unused) Drive 2 (512GB): Mirrors 512GB to Drive 1 (512GB) = 512GB used (0 unused) --> Total usable space (excluding overhead): still 512GB
January 15, 20251 yr You might be right with what you found, but I just read this stack exchange question from 2020 that might pertain to the usage too. https://superuser.com/questions/1582966/difference-between-filesystem-btrfs-size-with-df-btrfs-and-baobab
January 15, 20251 yr Author Solution I think that was related to an an unbalanced pool (which needed a btrfs balance to correct). I removed my 240GB drive from the pool. Unraid now reports usuable space on the pool of 511GB (formerly 512GB). Note that the pool size went down (from 589GB to 477GB), but used and available did not change whatsoever, despite removal of the 240GB drive ! : # btrfs fi usage -T /mnt/cache Overall: Device size: 953.88GiB Device allocated: 426.06GiB Device unallocated: 527.82GiB Device missing: 0.00B Device slack: 0.00B Used: 422.18GiB Free (estimated): 264.25GiB (min: 264.25GiB) Free (statfs, df): 264.25GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 264.42MiB (used: 0.00B) Multiple profiles: no Data Metadata System Id Path RAID1 RAID1 RAID1 Unallocated Total Slack -- --------- --------- --------- -------- ----------- --------- ----- 1 /dev/sdf1 211.00GiB 2.00GiB 32.00MiB 263.91GiB 476.94GiB - 2 /dev/sdj1 211.00GiB 2.00GiB 32.00MiB 263.91GiB 476.94GiB - -- --------- --------- --------- -------- ----------- --------- ----- Total 211.00GiB 2.00GiB 32.00MiB 527.82GiB 953.88GiB 0.00B Used 210.66GiB 444.53MiB 48.00KiB # df /mnt/cache Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdf1 500107080 221614288 277087344 45% /mnt/cache # df /mnt/cache -h Filesystem Size Used Avail Use% Mounted on /dev/sdf1 477G 212G 265G 45% /mnt/cache This is a big win from my perspective -- one less drive in the system with no downside. Also worth mentioning for anyone who finds this... the carfax calculator I linked to above is wrong (since it showed 0 unusable space) ! Edited January 15, 20251 yr by tcharron
January 15, 20251 yr Author 35 minutes ago, tcharron said: For some fun, I tried asking ChatGPT. I'm pretty sure it lied to me. It said I would only have 480GB of usable space and said that this is how my data would be allocated: By explaining how I thought this should work (as I outlined above), I was able to get ChatGPT to admit that it was wrong !
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.