BTRFS Cache Pool - Striping?

[GlitchNZ]

I am about to upgrade to V6 from (Currently using 5.0.6) and I am very much looking forward to using the new Cache Pool Feature. I am about to purchase a number of Samsung Pro SSD's for an initial cache pool (and an 8TB HDD), and due to the price I want to make sure I get the most cost effective size/quantity. I may add more later and I need to make sure I have a solution that scales correctly by meeting the following requirements:

 

• Initial Cache will need at least 500GB usable space as I can write this much data in a single day
  
• Adding more drives should increase this space so that I can scale the pool overtime if needed
  
• The Cache will need to initially handle files ~300Gb in size
  
• The maximum file size should also be able to scale over time
  

 

Looking through the current documentation it only says that it uses BTRFS's "unique twist on traditional RAID 1", not what this actually means.

 

Normally RAID 1 is just mirroring, so if I needed 500GB usable space I would have to buy 1TB worth of hard drives. Also, normally RAID1 does not do striping so the maximum file size would be the about the same size as the smallest disk in the pool.

 

If I assume standard RAID 1 then I would need 2x500GB drives to meet the above requirements. However, if BTRFS is genuinely different to RAID1 then maybe maybe the cheapest solution wold be something different (3x250GB for example).

 

I would also be interested to know if the RAID system used can be changed - for example, to RAID 5 or RAID 10 so that I can use the version of raid most suited for my needs.

 

I want to buy the SSD's (and an 8TB Seagate ) and preclear everything before I do the upgrade so that everything is all good to go. I'm looking forward to the combination of Super Fast SSD Cache with a stupidly cheap (and slow) 8TB parity, and I just want to make sure I get it right.

 

 

[jonp]

You can presently toggle between btrfs raid modes after you initialize your cache pool.  You do this by clicking on "cache" on the main tab, and under the balance section you'll see a command those into a text box that says mconvert=raid1 and dconvert=raid1.  Changes those to 5 or 10 and click balance to rebalance to that RAID type.  This is done while the array/pool are online and in use (beauty of btrfs).

 

Presently raid5/6 in btrfs are still considered more experimental to us because we haven't tested with them and they still seem to be going through quite a few code revisions.  Raid10 is probably safer than either of those, but anything other than raid1 at this point is use at your own risk until we state otherwise.

[GlitchNZ]

Thanks Jon,

 

My main concern is about future scalability, so the fact that different raid versions can/will be supported give me hope.

 

The main part of my question still stands though, does BTRFS do striping in RAID1? What exactly is it's version of RAID 1 and how does it differ from traditional RAID 1

[Whaler_99]

Just found this and got me to thinking, when I setup my Cache Pool, I used 4 x 500GB drives. But, based on what you have said Jon, is my solution, since I have made no changes, other than the default, actually only currently using two drives in RAID1 and not using the others?

I guess form what you are saying, I should really only be running with two drives at this point?

[trurl]

Just found this and got me to thinking, when I setup my Cache Pool, I used 4 x 500GB drives. But, based on what you have said Jon, is my solution, since I have made no changes, other than the default, actually only currently using two drives in RAID1 and not using the others?

I guess form what you are saying, I should really only be running with two drives at this point?

I've never tried more than 2 personally, but I think the way it is supposed to work you would have 1000GB mirrored. What does your webUI say for cache size? I have 2 x 120GB and it says my cache size is 120GB, which is what I would expect since the other 120 is a mirror. For 4 x 500GB I would expect 1000GB with the other 1000GB a mirror of that.

[Whaler_99]

Ya, I just checked and I have 500gb and not a 1tb, so definitely not raid 10. I think something like this needs to be made more clear in the GUI as well. And if it is only suggested to run raid 1 and not 5 or 10, make that clear.

[JorgeB]

Ya, I just checked and I have 500gb and not a 1tb, so definitely not raid 10. I think something like this needs to be made more clear in the GUI as well. And if it is only suggested to run raid 1 and not 5 or 10, make that clear.

 

Using default settings with 4 x 500GB drives you would get a btrfs raid1 4 drive cache pool with 1TB usable space, if you only see 500GB something is wrong, check "btrfs filesystem show:" on the cache webpage for the number of devices in use.

[trurl]

raid1 is the default and all I have ever tried. Are you manually setting it to raid10? What happens if you use raid1?

[Whaler_99]

Ya, I just checked and I have 500gb and not a 1tb, so definitely not raid 10. I think something like this needs to be made more clear in the GUI as well. And if it is only suggested to run raid 1 and not 5 or 10, make that clear.

 

Using default settings with 4 x 500GB drives you would get a btrfs raid1 4 drive cache pool with 1TB usable space, if you only see 500GB something is wrong, check "btrfs filesystem show:" on the cache webpage for the number of devices in use.

 

The issue as I see it, is that a true RAID1 set is only two drives. Any more than that and your are using some combination of Raid1+0 or Raid0+1, or if your are Intel, just call it Raid1E. From what I have seen and read, using more than two drives on setup, you still only get a two drive mirror. Then you manual have to run the command to rebalance it as raid10. But then there had been mention not to run the cache array as a raid5 or raid10 yet.

 

Here is what is on the Cache Page:

 

btrfs filesystem show:

Label: none  uuid: 8eaec535-5a88-45fc-a40c-86442c538cf5

Total devices 4 FS bytes used 480.81GiB

devid    1 size 465.76GiB used 277.03GiB path /dev/sdl1

devid    2 size 465.76GiB used 278.00GiB path /dev/sdo1

devid    3 size 465.76GiB used 277.03GiB path /dev/sdm1

devid    4 size 465.76GiB used 278.00GiB path /dev/sdp1

 

 

btrfs filesystem df:

Data, RAID1: total=552.00GiB, used=479.66GiB

System, RAID1: total=32.00MiB, used=96.00KiB

Metadata, RAID1: total=3.00GiB, used=1.15GiB

GlobalReserve, single: total=400.00MiB, used=0.00B

 

What I read from this, I have a 500GB data partiton....

[JorgeB]

What I read from this, I have a 500GB data partiton....

 

That means the pool is using ~550GB, should have about 450GB free.

 

BTRFS raid works different:

btrfs combines all the devices into a storage pool first, and then duplicates the chunks as file data is created. RAID-1 is defined currently as "2 copies of all the data on different devices". This differs from MD-RAID and dmraid, in that those make exactly n copies for n devices. In a btrfs RAID-1 on three 1 TB devices we get 1.5 TB of usable data. Because each block is only copied to 2 devices, writing a given block only requires exactly 2 devices to be written to; reading can be made from only one.

[Whaler_99]

What I read from this, I have a 500GB data partiton....

 

That means the pool is using ~550GB, should have about 450GB free.

 

BTRFS raid works different:

btrfs combines all the devices into a storage pool first, and then duplicates the chunks as file data is created. RAID-1 is defined currently as "2 copies of all the data on different devices". This differs from MD-RAID and dmraid, in that those make exactly n copies for n devices. In a btrfs RAID-1 on three 1 TB devices we get 1.5 TB of usable data. Because each block is only copied to 2 devices, writing a given block only requires exactly 2 devices to be written to; reading can be made from only one.

 

I understand what your saying with RAID1 BTRFS, pretty much the same as Windows Storage Spaces... but, when I look at the stats:

 

btrfs filesystem df:

Data, RAID1: total=552.00GiB, used=479.66GiB

System, RAID1: total=32.00MiB, used=96.00KiB

Metadata, RAID1: total=3.00GiB, used=1.15GiB

GlobalReserve, single: total=400.00MiB, used=0.00B

 

I read this as having a 552GB partition of which 479GB is used. The 479 used is correct, that does correspond to all the data on the cache pool. If I have 500GB still free, why doesn't it either show it or why does "Data, RAID1..." read the way it does? I'm not saying you are not right, just the output on the page in unRaid just seems confusing then.

 

[JorgeB]

I read this as having a 552GB partition of which 479GB is used. The 479 used is correct, that does correspond to all the data on the cache pool. If I have 500GB still free, why doesn't it either show it or why does "Data, RAID1..." read the way it does? I'm not saying you are not right, just the output on the page in unRaid just seems confusing then.

 

Agree that the btfrs stats are not very intuitive, also doesn't show free space, but in your case, since you’re using same size disks the main page cache pool stats should report both used and free space correctly, the free space can be incorrectly reported when using different size disks.

[JimPhreak]

Can I change the btrfs balance from raid1 to raid10 on my cache pool without losing the data currently on it?

[JorgeB]

Can I change the btrfs balance from raid1 to raid10 on my cache pool without losing the data currently on it?

 

Yes, unless something unexpected happens.

 

On cache page, go to balance status, and run:

 

-dconvert=raid10 -mconvert=raid10