SSDs as array drives question


Guest

Recommended Posts

6 minutes ago, SSD said:

 

Would Plex metadata create a significant write amplification issue?  Plex metadata files are tiny, but there are a zillion of them!

Most probably. But as long as they aren't constantly updated they probably can't combine and reach into TB-class disk wear.

Link to comment

Just an addendum.

 

While enterprise disks can perform at optimum speed without trim, they can still take advantage of trim to reduce the write amplification.

If a 128 kB flash block contains 64 kB erased data and 64 kB current data and the OS wants to rewrite the 64 kB of current data then the disk has two options

- if the 64 kB erased data has been reported through trim, then the drive can erase all content on that flash block while finding a suitable flash block for the new 64 kB write. So 64 kB of file writes results in 64 kB of flash writes.

- if the 64 kB erased data hasn't been reported, then the drive will pick a suitable spot for the 64 kB write. Then it will still think 64 kB data is valid on the original block. So when it later decides to erase that flash block, it first has to copy 64 kB to another block. So a single 64 kB file write results in 64 + 64 kB of flash writes. So without the trim, there was a larger write amplification.

 

Another thing to consider here is that unRAID, like SnapRAID, doesn't stripe.

 

So with 4+1 system with a traditional striped RAID 5, a 1 TB file write will result in 1.25 TB of flash writes since the parity adds 25% additional surface.

 

With unRAID, a 1 TB file write means the parity drive also has to handle a 1 TB write - so in total 2 TB of SSD writes.

 

In a traditional RAID 5, all drives gets the same amount of wear. You can select enterprise or normal drives depending on amount of writes the system needs to handle.

 

In unRAID, the parity drives would need to be expensive enterprise drives to be able to handle the additional number of TB of writes. Or the user would have to replace the parity drives 5-10 times more often depending on number of data disks.

 

One strong part about unRAID is that the unstriped data means only needed HDD has to be spinning. This is excellent for media consumers. With SSD this doesn't really matter because they don't produce noise and consume as much power which means that one of the unRAID advantages would no longer hold true. unRAID would still have a big advantage that each drive holds a valid file system so you can't get a 100% data loss because you lost one drive more than the number of parity drives.

 

In the end, I think it would be better if unRAID could add support for multiple arrays - mixed freely between unRAID arrays and normal RAID-0, RAID-1, RAID-5, ... than to focus on running a strict SSD main array. Especially if unRAID could still merge multiple arrays into unified user shares. That would allow people to select two-disc SSD RAID-1 for applications and special data while keeping the huge bulk data on much cheaper HDD.

  • Like 1
Link to comment
50 minutes ago, pwm said:

Especially if unRAID could still merge multiple arrays into unified user shares. That would allow people to select two-disc SSD RAID-1 for applications and special data while keeping the huge bulk data on much cheaper HDD.

I think I know what you mean in the big picture, but you do know you just described exactly to a T what currently is in place with the normal array + BTRFS RAID 1 cache that is set up by default if you add 2 SSD's to the cache pool, right?

 

Or were you being funny and making a point that we already have exactly what is needed without additions?

 

Sorry if I misinterpreted your intent, it just came across to me as amusing.

Link to comment
Just now, jonathanm said:

I think I know what you mean in the big picture, but you do know you just described exactly to a T what currently is in place with the normal array + BTRFS RAID 1 cache that is set up by default if you add 2 SSD's to the cache pool, right?

Yes, I know about the cache pool. But that's just a single additional mirror.

 

My main storage server is currently running 3 separate mirrors besides the main disk array. And I plan to add a fourth mirror and possibly a second full disk array when I continue to throw in more disks. The only way I can merge unRAID with the other functionality is if I run unRAID virtualized to handle a single array in the main server. Many separate mirrors means I can get huge, disk read/write speeds to keep a 10Gbit/s network happy streaming media concurrently with making larger backups.

 

So currently, I have multiple unRAID for secondary tier storage pools. Which also means I can't take full advantage of the Docker/VM support in unRAID because all the nice SSD mirrors are on the main server.

Link to comment
On 12/31/2016 at 11:50 PM, antaresuk said:

I read in the getting started guide that this isnt a good idea because of no TRIM support. What do people think of this, is this a big issue? Do others have all SSD as their array? Please forgive the newbie question  :)

 

I had a chat recently with the guys who develop all-flash NAS, they told me that there must be specific algorithms developed for this kind of media. Optimize writes to minimize wear out.

 

Besides you need specialized hardware, and honestly I don't think unRaid was built for speed

Link to comment

The way I am handling this is by having my SSDs setup in a btrfs RAID-0 cache pool, then syncing the information off onto the array for redundacy nightly. Of course, between syncs&movers the data on this cache pool is vunerable, but having 1TB of space available at SSD access speeds is worth it to me. I really only use it for cacheing and my Steam games library. So if you are just using it for your audio and video files, something like this might be a viable option for you.

Edited by ryoko227
Link to comment
  • 4 weeks later...
On 1/2/2017 at 1:59 PM, limetech said:

A file system issues TRIM commands as a result of deleting a file to tell the ssd that the set of blocks which previously made up the file are no longer being used.  The sdd can then mark those blocks as 'free'.  Later when the ssd internal garbage collection runs, then it knows that it doesn't have to preserve the contents of those blocks.  This makes garbage collection more efficient.  There are lots of articles that explain this.

 

The trouble this causes for parity-based array organizations is that the data returned from a TRIM'ed data block can be indeterminate.  This paper is a bit wordy but lays it out on p. 13:

 

 

 

To boil this down for unRAID: it should work to use SSD's in an unRaid P or P+Q array if TRIM is not used.  This is current behavior.  However note that:

a) Write performance can degrade faster on data disks depending on how many file deletions take place.

b) The parity disk is also written for each data disk write.

c) The data disks really should be completely written first because theoretically a block that was never written from the point of view of the SSD, can return non-deterministic data for those blocks.  We have not seen this happen, but then again we have not run too many SSD arrays (it would show up as parity sync errors).  This is pretty undesirable thing to do however since it will guarantee slowing down subsequent writes.

d) If you don't want to pre-write the disks as above, then only use SSD's that support "DX_TRIM" or "DZ_TRIM", and instead of writing the disks with zeros, simply use 'blkdisard' command to first TRIM the entire device instead.

 

You can use the 'hdparm' command to determine if your SSD's have this support:

 

 


hdparm -I /dev/sdX   # substitute X for your ssd device assignment
 

 

 

You want to look near the end of the "Commands/features:" section for:

 

          *    Data Set Management TRIM supported

 

Following this will either see this:

 

          *    Deterministic read data after TRIM

 

or you will see this:

 

          *    Deterministic read zeros after TRIM

 

or you won't see either of the above (if this is the case, do not use in unRAID P or P+Q array).

 

In a future release we do plan to add proper TRIM support to array disks.  Here's a heads-up on that.  In order to support TRIM in unRaid P or P+Q array, we must add code to the md/unraid driver and all SSD's in the array must support either "DX_TRIM" or "DZ_TRIM" mode as described above.  In addition there's a really good chance we will only support SSD's that support "DZ_TRIM" since to support "DX_TRIM" is a lot more work  ;-)

 

Thanks for this post.  Right now I have one SSD in my array that has been there for years and has not been at all problematic.  It uses "Deterministic read data after TRIM."  However, this drive is getting too small for my needs and I want to get a larger drive now, so naturally I want to make sure it is compatible with unRAID TRIM in the future. 

 

If only DZ_TRIM is going to be supported, I want to find a disc that supports this.  I however can't figure out a way to discover what type of TRIM any given SSD supports short of buying it and putting it into a Linux machine and checking.  Is there a list or any reference available to figure out what SSD drives support DZ_TRIM TRIM?  That said, there are also several hardware versions of the 840 PRO so who knows if they all handle TRIM the same way?!?

 

I read a random post in some random Linux forum that the Samsung EVO 840 PRO supports DZ_TRIM but that the 850 does not.  That is literally all the info I can find about SSD drive trim types after a lot of Googling.  Anyone have any guidance?

 

Thanks,

craigr

  • Like 1
Link to comment
  • 1 year later...
On 1/2/2017 at 2:59 PM, limetech said:

A file system issues TRIM commands as a result of deleting a file to tell the ssd that the set of blocks which previously made up the file are no longer being used.  The sdd can then mark those blocks as 'free'.  Later when the ssd internal garbage collection runs, then it knows that it doesn't have to preserve the contents of those blocks.  This makes garbage collection more efficient.  There are lots of articles that explain this.

 

The trouble this causes for parity-based array organizations is that the data returned from a TRIM'ed data block can be indeterminate.  This paper is a bit wordy but lays it out on p. 13:

 

 

 

To boil this down for unRAID: it should work to use SSD's in an unRaid P or P+Q array if TRIM is not used.  This is current behavior.  However note that:

a) Write performance can degrade faster on data disks depending on how many file deletions take place.

b) The parity disk is also written for each data disk write.

c) The data disks really should be completely written first because theoretically a block that was never written from the point of view of the SSD, can return non-deterministic data for those blocks.  We have not seen this happen, but then again we have not run too many SSD arrays (it would show up as parity sync errors).  This is pretty undesirable thing to do however since it will guarantee slowing down subsequent writes.

d) If you don't want to pre-write the disks as above, then only use SSD's that support "DX_TRIM" or "DZ_TRIM", and instead of writing the disks with zeros, simply use 'blkdisard' command to first TRIM the entire device instead.

 

You can use the 'hdparm' command to determine if your SSD's have this support:

 

 


hdparm -I /dev/sdX   # substitute X for your ssd device assignment
 

 

 

You want to look near the end of the "Commands/features:" section for:

 

          *    Data Set Management TRIM supported

 

Following this will either see this:

 

          *    Deterministic read data after TRIM

 

or you will see this:

 

          *    Deterministic read zeros after TRIM

 

or you won't see either of the above (if this is the case, do not use in unRAID P or P+Q array).

 

In a future release we do plan to add proper TRIM support to array disks.  Here's a heads-up on that.  In order to support TRIM in unRaid P or P+Q array, we must add code to the md/unraid driver and all SSD's in the array must support either "DX_TRIM" or "DZ_TRIM" mode as described above.  In addition there's a really good chance we will only support SSD's that support "DZ_TRIM" since to support "DX_TRIM" is a lot more work  ;)

 

I can see this thread is quite old and I am wondering if there may be some updated information regarding how SSD's are handled in an array. Currently I attempt to run this command to determine the style of TRIM my devices are using and the data is limited. 

 

I can only assume this has to do with the built in HP controller I am using. I have attempted to switch the SMART controller type from "Automatic" to the HP Specific type of controller listed but this does not help. 

 

I fear that I am unable to determine this information due to the RAID controller blocking the query.  

 

Is there a vendor or an SSD controller type list that might be able to be referenced before purchasing SSD's for an array? 

 

 

This is currently the only information I get back from the hdparm command:

 

/dev/sdb:
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ATA device, with non-removable media
Standards:
        Likely used: 1
Configuration:
        Logical         max     current
        cylinders       0       0
        heads           0       0
        sectors/track   0       0
        --
        Logical/Physical Sector size:           512 bytes
        device size with M = 1024*1024:           0 MBytes
        device size with M = 1000*1000:           0 MBytes 
        cache/buffer size  = unknown
Capabilities:
        IORDY not likely
        Cannot perform double-word IO
        R/W multiple sector transfer: not supported
        DMA: not supported
        PIO: pio0 

 

Any information is appreciated. 

 

Thanks so much!

Link to comment
  • 2 years later...
On 1/2/2017 at 8:59 PM, limetech said:

To boil this down for unRAID: it should work to use SSD's in an unRaid P or P+Q array if TRIM is not used

I know it's an old post, but I need to ask as this is the only official statement regarding SSDs in the array. Why did you wrote "should"?

 

And why this sentence:

On 1/2/2017 at 8:59 PM, limetech said:

or you won't see either of the above (if this is the case, do not use in unRAID P or P+Q array).

 

This is confusing as it is not possible to TRIM SSDs in the array, why should they need to support any TRIM feature?

 

Regarding my own experience (I build several SSD only Unraid servers for different customers), there is absolutely no reason, not to use SSDs in the array.  @JorgeB seems to be the only person who had bad experiences because of an defective Kingston V300 which changed data while his server was shutdown (I mean an SSD which changes data while it's powerless can't be ok and it has clearly nothing to do with TRIM or not to TRIM). And until now he never heard about other people with similar problems.

 

So why are SSDs in the array still not officially "approved"? I think its time to take away the fear.

Link to comment
3 minutes ago, mgutt said:

why should they need to support any TRIM feature?

IIRC that was about enabling TRIM support for the array, at this time and since TRIM is not supported you can use any SSD, well except the V300 due to unrelated issues, AFAIK it's the only model that should be avoided and it's been discontinued for years now so likely not a problem anyway.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.