BtrFS SSD drives in RAID 0 - will fstrim work?

[bubbaQ]

I have been running 4 SSDs in hardware RAID 0 as my cache drive for a few years.  I need it since I have 10 GBE between workstation and unRAID and really need the speed as I work with large (>100GB) files regularly.  I also back up large drives to unRAID regularly over 10 GBE and need the speed.

 

I leave “hot” files on the cache, since I need read access as fast as possible for a couple of weeks while I work with them.

I have good performance, but the drives are “hidden” behind the HW RAID controller.  Most of the files are working copies of files I have stored offline, so they just sit on cache till I am done with them, and they never get moved to the array. 

 

As expected, performance decreases over time since I can’t trim the RAIDed drives.  I have to back up cache, break the RAID, secure erase the SSDs, reconfig the RAID and then restore.

 

I need a much larger cache drive so I am going to use four 4TB SSDs in RAID 0.  But I am thinking about using BtrFS to do the raid instead of the HW contoller.  The deciding factor will be if I can use fstrim to trim the drives RAID'ed with BtrFS w/o breaking/rebuilding the RAID.

 

There is conflicting info out there as to whether you can do effective fstrim on the drives in a BtrFS RAID 0 configuration.  Some say you can do it but it won’t actually be effective, and other say it does work as expected.

 

Anyone here want to chime in with their $0.02?

 

[JorgeB]

AFAIK it works fine, there were some fstrim fixes late last year or earlier this year related to trimming all the unallocated space, but that affects all raid profiles, the only profile I'm not sure trim works well is raid5/6.

[bonienl]

I have a cache pool of four SSD devices. All connected directly on the SATA ports of my motherboard.

The pool is using BTRFS and set up as RAID10, trim works.

[John_M]

54 minutes ago, bonienl said:

All connected directly on the SATA ports of my motherboard

I believe this is significant. It's mentioned in some threads that SSDs connected to certain HBAs don't get trimmed.

[JorgeB]

4 minutes ago, John_M said:

It's mentioned in some threads that SSDs connected to certain HBAs don't get trimmed.

Yes, most currently recommended LSI HBAs don't currently support trim with Unraid, but the user gets an error when running fstrim, so easy to know, what I believe bubbaQ is asking is if the fstrim command reports the pool was trimmed but it's not actually trimmed because of some btrfs issue.

[bonienl]

6 minutes ago, johnnie.black said:

if the fstrim command reports the pool was trimmed but it's not actually trimmed because of some btrfs issue

Not sure what you mean with this.

# fstrim -v -a
/etc/libvirt: 920.5 MiB (965246976 bytes) trimmed
/var/lib/docker: 14 GiB (14971002880 bytes) trimmed
/mnt/cache: 635.4 GiB (682271277056 bytes) trimmed

Here cache is a pool of four SSD devices.

[JorgeB]

7 minutes ago, bonienl said:

Not sure what you mean with this.

I may have misunderstood the question, but by this:

11 hours ago, bubbaQ said:

Some say you can do it but it won’t actually be effective, and other say it does work as expected.

I took it as he was asking if a raid0 pool is being trimmed and fstrim reports success but not working as it should.

[bubbaQ]

On 12/1/2018 at 8:13 AM, johnnie.black said:

I took it as he was asking if a raid0 pool is being trimmed and fstrim reports success but not working as it should. 

That is the correct interpretation.

 

On 12/1/2018 at 7:57 AM, johnnie.black said:

Yes, most currently recommended LSI HBAs don't currently support trim with Unraid, but the user gets an error when running fstrim, so easy to know, what

IIRC, LSI interrogates the drive capabilities, and if a drive does not specify the type of trim support (DRAT or RZAT)  it will not pass trim commands to the drive.  Check an SSD with hdparm -I /dev/sdX.  The Sammy 860 EVO's I am using correctly report RZAT. 

[JorgeB]

6 hours ago, bubbaQ said:

IIRC, LSI interrogates the drive capabilities, and if a drive does not specify the type of trim support (DRAT or RZAT)  it will not pass trim commands to the drive.  Check an SSD with hdparm -I /dev/sdX.  The Sammy 860 EVO's I am using correctly report RZAT. 

It used to work, but it stopped working on newer releases, even with SSDs supporting RZAT

[bubbaQ]

Well, I'm glad I asked.  My existing cache drive was 4 PNY 256GB SSDs.  hdparm -I says they are RZAT:

 

  Data Set Management TRIM supported (limit 8 blocks)
  Deterministic read ZEROs after TRIM

 

So I decided to do some testing, took them out of HW RAID, secure erased them, and set them up in a btrfs RAID0 cache pool.  Worked fine, RAID0 was confirmed working.

 

But fsutil refused to run on them... reported "discard operation not supported."

 

Had I not started this thread, I probably would not have proceeded.  But in spite of that test, I went ahead and tried a btrfs RAID0 cache pool with 2 Sammy 4TB 860 EVO SSDs. 

 

The kicker.... fstrim appears to work on the Samsung array (at least no errors).  Before you ask, the Samsung drives are on the same controller (in fact same ports) as the prior cache pool of PNY SSDs.  But I cannot see any relevant difference in trim support between the PNY and Sammy SSD's

 

[JorgeB]

Interesting, so trim still works with the 860 EVOs, I'm assuming with a current Unraid, what model LSI do you have?

[bubbaQ]

On 12/3/2018 at 1:53 PM, johnnie.black said:

Interesting, so trim still works with the 860 EVOs, I'm assuming with a current UnRAID, what model LSI do you have? 

Don't touch LSI.  I use either Areca or Adaptec SAS controllers with HW RAID and onboard cache.  This server is an Adaptec 7000Q series.

 

I'm staying on 6.5.3 ... can's stand the UI in 6.6... makes me want to vomit.