btrfs cache pool


limetech

Recommended Posts

So jonp, I have not had a chance to mess with this but I was wondering if I could perhaps do this via the command line manually? Kind of like what was done initially for cache pool for unRAID v6 beta 6.

 

I probably wont be playing with this for another week or two but figured I would ask the question now.

 

Yes, you can create BTRFS pools from command line to experiment with if you'd like.

Link to comment

So jonp, I have not had a chance to mess with this but I was wondering if I could perhaps do this via the command line manually? Kind of like what was done initially for cache pool for unRAID v6 beta 6.

 

I probably wont be playing with this for another week or two but figured I would ask the question now.

 

Yes, you can create BTRFS pools from command line to experiment with if you'd like.

Cool!!  I figured that would be the answer but wanted to make sure before venturing down that path.

Link to comment
  • 3 weeks later...

Might be a dumb question, but can I take my unRAID created cache pool (RAID1) and run the following command on it to convert the data portion to RAID0?

 

        btrfs balance start -dconvert=raid0 /mnt/cache

 

Actually, I might just shutdown the array first and mount the cache at a temp directory before I do this. 

 

But assuming I convert to raid0 (and it seems like from reading some docs on btrfs, this would work), would Unraid still be ok with the cache pool?  I've seen Unraid running btrfs commands when bringing up the array, and what worries me is, does it make any assumptions about the pool type in the cache and potentially screw up the configuration?

Link to comment
  • 1 month later...

Might be a dumb question, but can I take my unRAID created cache pool (RAID1) and run the following command on it to convert the data portion to RAID0?

 

        btrfs balance start -dconvert=raid0 /mnt/cache

 

Actually, I might just shutdown the array first and mount the cache at a temp directory before I do this. 

 

But assuming I convert to raid0 (and it seems like from reading some docs on btrfs, this would work), would Unraid still be ok with the cache pool?  I've seen Unraid running btrfs commands when bringing up the array, and what worries me is, does it make any assumptions about the pool type in the cache and potentially screw up the configuration?

I had tried this and it didnt work; best we wait for the feature to be implemented by lime-tech :)
Link to comment
  • 2 weeks later...

I don't quite understand how this is supposed to work, even after running btrfs balance. It looks like I have all data on one drive, the largest disk. Isn't it supposed to work like raid1 for redundancy and speed? What happens if I loose one disk?

 

I would like to se a raid1+0 for stripe and redundancy either as a cache pool, or as a second array for my VMs.

 

unraid_btrfs.jpg

Link to comment
  • 3 months later...
  • 1 year later...

Is there any way for the cache pool to be a READ cache as well as a write cache, or even just a read cache? I read the same or predictable/sequential data off the array over long periods of time, and it would be great to be able to spin down the HDD and just have the last-accessed files in a cache on the SSD cache pool, or even in memory. I think this is considered "bcache" in some Linux Distros. Has this ever been discussed? Thanks!

 

Link to comment
50 minutes ago, kubed_zero said:

Is there any way for the cache pool to be a READ cache as well as a write cache, or even just a read cache? I read the same or predictable/sequential data off the array over long periods of time, and it would be great to be able to spin down the HDD and just have the last-accessed files in a cache on the SSD cache pool, or even in memory. I think this is considered "bcache" in some Linux Distros. Has this ever been discussed? Thanks!

 

 

Discussion:

 

Or just put the files into a cache only share if that's feasible since you're running a pool, its already redundant.

Link to comment
  • 4 weeks later...
On 7/17/2017 at 9:15 AM, Squid said:

Or just put the files into a cache only share if that's feasible since you're running a pool, its already redundant.

 

 

Thanks for the link! That was a rabbit hole :D What I got out of that was that if the SSD supports automatic/behind the scenes trim/garbage collection, it's safe to use in the array. Then it's just a matter of storing files on that drive and moving them to slower storage when I run out of space. 

Link to comment
1 hour ago, kubed_zero said:

Thanks for the link! That was a rabbit hole :D What I got out of that was that if the SSD supports automatic/behind the scenes trim/garbage collection, it's safe to use in the array. Then it's just a matter of storing files on that drive and moving them to slower storage when I run out of space. 

I didn't review the posts in the "rabbit hole" but I think the concern with using SSD in the parity array is precisely because it might invalidate parity when the SSD did anything "behind the scenes".

Link to comment
1 hour ago, trurl said:

I didn't review the posts in the "rabbit hole" but I think the concern with using SSD in the parity array is precisely because it might invalidate parity when the SSD did anything "behind the scenes".

As I understand it, "doing things behind the scenes" can occur in two different ways. 1, the trim/garbage collection command is issued by the OS, and the drive goes and moves blocks around to get to an ideal performance state. If the OS doesn't support this or the commands are blocked by an intermediate layer like hardware RAID or unRAID, TRIM can't occur and the drive's performance will decay over time. 

 

The second option is if the drive does garbage collection behind the scenes without informing the OS. Then, I don't think there are any changes observed by the system before/after a garbage collection occurs, since it is abstracted away entirely into the SSD's hardware. 

 

With this second scenario then, it should be safe to run SSDs alongside HDDs since they will behave identically from the perspective of the OS.

 

Am I understanding that right?

Link to comment
2 minutes ago, kubed_zero said:

The second option is if the drive does garbage collection behind the scenes without informing the OS. Then, I don't think there are any changes observed by the system before/after a garbage collection occurs, since it is abstracted away entirely into the SSD's hardware. 

 

With this second scenario then, it should be safe to run SSDs alongside HDDs since they will behave identically from the perspective of the OS.

What you must keep in mind is that all bits on an unraid drive are sacred, not just the ones assigned to a specific file. If the SSD takes it upon itself to zero the content of a deleted file, then when unraid needs those bits to reconstruct a different drive, they will be zero instead of the way unraid last knew about them. The only way background operations can work with unraid is if every bit in the partition is still identical after the SSD is done with its operation. Typically SSD's will use unpartitioned space to do their background stuff, but unraid expands the partition to fill the entire drive by default, so no slack space.

 

8 minutes ago, kubed_zero said:

If the OS doesn't support this or the commands are blocked by an intermediate layer like hardware RAID or unRAID, TRIM can't occur and the drive's performance will decay over time.

That is exactly the situation right now in unraid. Because of what I explained above, you can't enable trim on an array member SSD, so after a period of time the SSD will slow dramatically.

  • Upvote 1
Link to comment
  • 3 months later...

There a two ways for a SSD to get access to pre-erased flash blocks to handle writes.

 

The important thing to remember is that the SSD does not understand any operating system, and have no concept of files. But it virtualizes a sector-based interface to the OS, and performs on-the-fly remappings between sector addresses and actual flash blocks.

 

One way for a SSD to erase unused flash space is that the OS automatically on file delete or manually by the user running a program sends down TRIM information to the SSD where it mentions which logical sectors in the file system that are currently unused. The memory controller in the SSD translates these sector numbers into locations on flash blocks. The flash blocks are much larger than the virtual sectors it reports to the OS. But when the SSD performs trim it results in changed content for some of the sectors that the OS can see. This creates havoc for a multi-disk system with parity, since the RAID code will not know about the changes - unless the RAID code intercepts the TRIM list. Besides having issues when used with parity, this kind of disk also gets very bad write performance when it is full or nearly full, because there are no free flash blocks to use. All new writes must first find a flash block to erase before the file changes can be written.

 

But commercial grade SSD have overprovisioning. So a 100GB SSD may have 110 GB actual flash data. Whenever the OS issues writes, the flash controller picks up free flash blocks out of the 10GB extra flash area and remaps into the file system range. It then issues flash block erase of the previously used blocks and moves them to the free list.

 

This means that there is no need to send TRIM data to the SSD. It always has a pool of erased blocks in the bag of over-provisioning sectors. The OS will never see any additional disk changes so no parity is broken. And even when the disk is 100% full it can still handle a big rewrite without bad write speeds because the disk never has to wait for flash blocks to be erased before it performs new writes. It just picks up erased sectors from the overprovisioning.

Link to comment
On 8/13/2017 at 10:34 PM, jonathanm said:

That is exactly the situation right now in unraid. Because of what I explained above, you can't enable trim on an array member SSD, so after a period of time the SSD will slow dramatically.

As I mention in my previous post - a SSD with overprovisioning can continue to give good performance despite never seeing any TRIM requests. Any flash block that gets filled is matched by a flash block that gets erased and returned to the free pool, waiting for a future write to need it.

Link to comment
  • 1 month later...
On 2/21/2015 at 10:23 AM, jonp said:

 

Hmm, I get the desire for maximum speed, but are we talking purely about writing data to the array over the network?  If so, you will gain nothing by using RAID0 for cache.  You also wont' gain anything by using an SSD over an HDD for the cache in that scenario.  Where SSDs really win is in 3 key areas:

 

1)  No spin up delays and less latency.

2)  Docker applications performing IO intensive tasks will perform better (this is subjective).

3)  Virtual Machines (especially localized ones) will see a massive performance increase.

 

#3 is the only one that I'd expect to see a performance gain with RAID0 on (potentially), but even that's arguable.  If you're not using containers or VMs (just using the cache for it's network write-speed increase), the bottleneck will be your network connection which on a 1gbps link is capped at 125MB/s (within performance capabilities of HDDs).

 

Just trying to understand the desire / use case for RAID0.

The reason i'm interested in this is because I use a 10g LAN card in both my server and my PC this is where the RAID0 wil make a difference for me.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.