BTRFS Cache Pool - available space


Recommended Posts

I was going to post this in defect reports but figured I would start here since I don't know if there is an issue or not.

 

unRAID v6rc4

 

I have 3x 128GB SSDs in a BTRFS cache pool that I use for Docker confguration and VM image storage:

 

A9xOsVp.jpg

 

In a quick chat with jonp in this thread, we both assumed that I should have 256GB of usual storage space (1 SSD sacrificed for protection):  http://lime-technology.com/forum/index.php?topic=40354.msg379861#msg379861

 

Then I started to doubt myself and did some quick checking.  The simplest way I could think of to check used space would be to highlight all files/folders on my cache share and get the properties.  I know that this is not exact science but I figured it would be close enough.  Here is what it came back with:

 

tV3bgJU.jpg

 

According to the above, I am using only 65GB of space but yet according to unRAID I have only 87.4GB of free space.

 

Mucho confused!  can someone shed some light?

 

John

Link to comment

Here's what I see, but I'm not an expert here.  The first picture shows what appears to be a RAID1 array, with 3 128GB SSD's, so their total size would be 128GB, with triple redundancy.  If 40.6GB is used, then 87.4GB free would be correct.

 

The second picture shows Windows view of that RAID1 device, and since it does not know about BTRFS COW, it assumes a large used figure of 48.2GB, what would be used if it was on a Windows drive.  The BTRFS Copy On Write deduplication has apparently saved you almost 8GB in duplicated storage.  Then Windows tries to guess at the size on disk, and it uses what it *thinks* the cluster size is, which if wrong will produce wildly wrong figures.  Since you have over 30,000 files, many of them are probably very small, which with a large cluster size, leaves a lot of unused space in the last or only cluster for each file.  That explodes the 48.2GB to 65.0GB.

Link to comment

Agree with Rob => I think the array is set for triple redundancy, and the numbers shown on the UnRAID display are correct.    The Windows display is incorrectly reporting a much higher "Size on Disk" number, since it doesn't know the actual allocation scheme used in the btrfs pool.

 

I don't know if there's a way to configure the pool to have less redundancy but more space ... but if so, I'd still only expect to see 192GB of space, since can't replicate 256GB on a 128GB drive  :)    It COULD,  if the configuration option is available, ensure all data was replicated on a different drive while still providing 192GB of storage with those 3 drives.

 

Link to comment

192GB is what I would have expected, based on what has been discussed with BTRFS usage, but it would not be a true RAID1.  Perhaps it should be called RAID-B, because it's a special BTRFS variation.  It's my sense that 'RAID-B' creates a RAID0 of all the physical volumes, then mirrors the top half with the bottom half.  That would give you a redundant 192GB in this case.

Link to comment

192GB is what I would have expected, based on what has been discussed with BTRFS usage, but it would not be a true RAID1.  Perhaps it should be called RAID-B, because it's a special BTRFS variation.  It's my sense that 'RAID-B' creates a RAID0 of all the physical volumes, then mirrors the top half with the bottom half.  That would give you a redundant 192GB in this case.

 

Top half with bottom half wouldn't guarantee the mirrored data was on different disks.  I believe what it actually does is effectively create 3 RAID-1 volumes that each contain 1/2 of one drive and 1/2 of another ... and then combines these into a single logical pool.    This was all data is stored on two different drives.

 

But regardless of the mechanics, I'd expect 192GB of protected pool storage with 3 128GB drives.

 

Link to comment

So I was wrong on your free space.  It should be only 192GB usable capacity.  BTRFS RAID 1 needs to be able to write each block of data to two different devices so that in case any one single device were to fail, 100% of data is still accessible.  To do this across three devices will require that each device carries 1/2 of it's own data, 1/2 data from the other two devices.  So 3 x 128GB devices would equate to 192GB of usable capacity in a BTRFS RAID 1 group.

 

Now as far as trying to manually calculate the capacity, you'll go insane trying to do that by hand.  This is due to the fact that BTRFS doesn't simply store data, but also uses metadata and a global reserve.

 

Your pool should be able to store up to 192GB of total data, but I think we could probably do a better job of showing that in the webGui.  Will look into this.

Link to comment

The pool information on the first cache link will give you a better idea of what is actually being used. Click your first cache link then scroll down to the pool information.

9e7cc3413669354.jpg

 

It will look something like the following, but you will have three disks.

Label: none  uuid: a4a6e12a-79d2-4647-a001-6b3cb6d35389
Total devices 2 FS bytes used 166.25GiB
devid    1 size 1.82TiB used 168.03GiB path /dev/sdj1
devid    2 size 1.82TiB used 168.03GiB path /dev/sdk1

btrfs-progs v4.0.1

 

If all your disks have the same amount used, it might be, because your btrfs pool hasn't been balanced yet.

 

Have you attempted to balance your cache pool?

 

https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

 

If you go down to the section "Adding new devices". It talks about it.

At this point we have a filesystem with two devices, but all of the metadata and data are still stored on the original device(s). The filesystem must be balanced to spread the files across all of the devices.

btrfs filesystem balance /mnt

 

You can balance in the GUI.

Click your first Cache drive.

9e7cc3413669354.jpg

 

Go down to balance status. Then click the balance.

26566e413669352.jpg

 

After you balance it should allocate data correctly. Or, at least I am going under that assumption.

 

 

You can also do the balance manually according to the docs.

https://wiki.archlinux.org/index.php/Btrfs

 

This is down further by RAID features

The raid level can be changed while the disks are online using the btrfs balance command:

# btrfs balance start -mconvert=RAIDLEVEL -dconvert=RAIDLEVEL /path/to/mount

For 2 disk sets, this matches raid levels as defined in md-raid (mdadm). For 3+ disk-sets, the result is entirely different than md-raid.

For example:

Three 1TB disks in an md based raid1 yields a /dev/md0 with 1TB free space and the ability to safely lose 2 disks without losing data.

Three 1TB disks in a Btrfs volume with data=raid1 will allow the storage of approximately 1.5TB of data before reporting full.

 

So for unraid it would be:

btrfs balance start -mconvert=raid1 -dconvert=raid1 /mnt/cache

 

Like I said above though. Using the balance button in the webGUI should handle it.

 

cache_click.png.f59bc15875e801633bc2ebb0097ca7d8.png

balance_click.png.88e01f22e35fe6e8a16f1adddb53b0c4.png

balance_click.png.95053e0c22938c67cfc087e0fdba9c25.png

Link to comment

What do you make of this:

 

Label: none  uuid: f5497a6f-06d0-4702-b941-f18cd1acac53
Total devices 3 FS bytes used 37.79GiB
devid    1 size 119.24GiB used 119.19GiB path /dev/sdb1
devid    2 size 3.97MiB used 0.00B path /dev/sdc1
devid    3 size 119.24GiB used 119.19GiB path /dev/sdd1

btrfs-progs v4.0.1

 

Honestly, the above makes no sense to me whatsoever.  According to it, I have .05GB + 3.97MB +.05GB of free space.  :)

 

I'm starting to wonder if the RAID  set was not properly created when I inially jumped into v6 (b10 maybe?  I would need to check).

Link to comment

Can you run this on your system and see what you get?

 

btrfs filesystem df -H /mnt/cache

 

Output should be something like the following.

 

root@NAS:~# btrfs filesystem df -H /mnt/cache
Data, RAID1: total=178.24GB, used=177.57GB
System, RAID1: total=33.55MB, used=49.15kB
Metadata, RAID1: total=2.15GB, used=936.03MB
GlobalReserve, single: total=318.77MB, used=0.00B

 

I'm wondering if they metadata might be eating up the additional space. Note, you won't see free space. It really just shows what is being used by data verse metadata.

 

Also, yeah... Your second device doesn't look right at all. The issue might be with it. btrfs might be having issues with seeing the device as a device with the full amount of data available to it. So, it can't balance the data across three disks, because it thinks your second disk is 4 megs inside.

Link to comment

root@unRAID:~# btrfs filesystem df -H /mnt/cache
Data, RAID1: total=126.87GB, used=40.50GB
System, RAID1: total=33.55MB, used=49.15kB
Metadata, RAID1: total=1.07GB, used=77.82MB
GlobalReserve, single: total=33.55MB, used=0.00B

 

I always thought that 2nd disk looked funky but figured it must just be some artifact from the RAID.  Check the below (partition format) for cache2 vs. cache3...

 

ljO3dmA.png

 

UE42rMy.png

Link to comment

I have the same thing with the partition formats. I don't think that is an issue.

 

Ok, so going back to one of my questions. Did you attempt to do a balance (click the balance button on the cache page)?

 

If you do a balance it should reread the metadata, and set things so they dedupe already written extra copies of the data. The balance rereads all data, and metadata, and writes it back out.

 

Link to comment

Finished...although it says this once it finsihed:

 

No balance found on '/mnt/cache'

 

 

Here is the device list:


Label: none  uuid: f5497a6f-06d0-4702-b941-f18cd1acac53
Total devices 3 FS bytes used 37.82GiB
devid    1 size 119.24GiB used 42.03GiB path /dev/sdb1
devid    2 size 3.97MiB used 0.00B path /dev/sdc1
devid    3 size 119.24GiB used 42.03GiB path /dev/sdd1

btrfs-progs v4.0.1

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.