Jump to content
dmwilson1990

SSD only RAID?

24 posts in this topic Last Reply

Recommended Posts

I've recently purchased the majority of components (all but GPUs) required to build a solid unRAID NAS / gaming machine. I would like to install six 500GB SSDs and stay away from platter based media if at all possible. However, unRAID's documentation states the following:

 

Do not assign an SSD as a data/parity device.  While unRAID won’t stop you from doing this, SSDs are only supported for use as cache devices due TRIM/discard and how it impacts parity protection.  Using SSDs as data/parity devices is unsupported and may result in data loss at this time.

 

As this will be a also NAS I really want something that's going to be rock solid stable, however, I've not touched a platter based drive in years. Is this outdated information or is unRAID really not capable of a SSD only RAID? Any advice would be greatly appreciated.

 

 

 

##########

- Intel Core i7-5820K 3.3GHz 6-Core Processor

- ASRock X99 Extreme4 ATX LGA2011-3 Motherboard

- Corsair Vengeance LPX 16GB (8 x 8GB) DDR4-3000 Memory

- 6x Samsung 850 EVO-Series 1TB 2.5" Solid State Drive

- 2x NVIDIA GTX 1080 Founders Edition (to be purchased on launch)

 

Share this post


Link to post

unRAID distinguishes between several different storage classes.  The parity protected storage array is typically composed of multiple high capacity spinners and used to store personal data and media.  SSDs are not appropriate for the storage array, either as data drives or parity drives.

 

The unRAID cache drive/pool can actually be a RAID 1'ish redundant array using BTRFS pooling.  SSDs are appropriate for the cache system.  Docker images, data, and VM images are typically stored on the cache pool. 

 

Finally unRAID supports devices mounted outside the data/cache array - these devices can be SSDs for VMs, etc.

Share this post


Link to post

Hmm...that does not make a lot of sense. I deal with a lot of servers at work (all hardware RAIDs) and every drive is an SSD; I guess I don't understand why RAID 5 is not possible with SSDs in a software RAID.

 

I can't bring myself to purchase a platter based drive. Would you recommend I place all six SSDs in the cache instead to avoid data loss?

Share this post


Link to post

SSDs on the array are not officially supported, doesn't mean they can't be used, trim won't work and there’s a small possibility that on some ssds, internal garbage collection could invalidate parity and compromise a disk rebuild, all evidence as of now is that that is not an issue, many users have 1 or more ssds on the array, I've been using a 24 ssd only array for some time with no issues, I do recommend doing frequent parity checks in the beginning and only continue using them if there aren't unexpected sync errors.

Share this post


Link to post

I figured I would not be the only one not using traditional platter based drives. Glad it's working well for you!

 

If I were to create an all SSD array what settings would you recommend I change to create a best case scenario for reliability? I know that unRAID supports btrfs which should support TRIM; however, is that only avabialible on the cache? Also, would dual M.2 SSDs as a cache for a slower SSD array be advantageous?

Share this post


Link to post

I use xfs, trim won't work on the array, no matter the file system used, it was disabled by lLT as it would almost certainly invalidate parity, it works on cache, xfs or btrfs.

Share this post


Link to post

DO NOT USE SSD as a array drive in UNRAID until Limetech say it is ok.

 

I have been advocating this is bumped up the testing list but it is unlikely to be done now until post 6.2.

 

Share this post


Link to post

SSDs are getting cheap enough that they're nearly useful for bulk data storage.  In the next year, I suspect we'll be seeing very affordable 2TB+ SSDs.

 

This feature is fast becoming a priority.

Share this post


Link to post

From how I see it, Its not really something that can be worked on by LimeTech since the drives tend to scramble themselves at a sector level without letting the OS know. I dont see how parity at a sector by sector level can ever be accepted as valid so it has to be switched to a file by file level given that behind the scenes activity.

Share this post


Link to post

If it truely is unsolvable then so be it but we will need to move from a snigle line hidden away in the manual to an active rejection of SSD drives when joining the array.

 

Data protection trumps every other features always.

 

But I cannot believe it is unsolvable. SSDs are here to stay and it will not be long, as HellDiverUK points out, when people will be buying SSDs more than spinners.

Share this post


Link to post

I've been using a SSD only server for a while, already did many rebuilds for size upgrades and never any any issue.

 

Trim would probably be a problem but is not an issue since it's disable on array devices and the internal garbage collection appears to be harmless, in the beginning I did daily parity checks, now do them weakly, never had a single parity sync error, file integrity plugin also does weekly scans and never found any cheksum errors.

 

There are others users with a single SSD on the array and AFAIK also never reported any sync errors, if there were any issues they would be detected by a parity check, that's why I recommend doing them frequently when using SSD(s) in the array, still since it's unsupported use them at your own risk, for now I only store recent data, like currently airing TV shows, that I can easily get again if needed.

 

 

Share this post


Link to post

Building an all-flash array is something we plan on experimenting with in the future.  For now, we don't have any official support for it other than allowing you to do it in the webGui.  Bottom line:  your mileage may vary.  It all comes down to the fact that we can't officially support what we don't test ourselves.  We also think there may be a number of tuning changes we would need to make to get the most of out SSD array devices, but then again, there are some users in this forum that have an all flash array already and report pretty decent performance.

Share this post


Link to post

Any update on this? Almost 3 years later, an all SSD array is more realistic than ever. Can we expect unraid to support this anytime soon officially?

Share this post


Link to post

Not sure how you come to the conclusion that an all SSD is more realistic, for who? SSD's beyond 2TB are quite expensive and if I want a storage array greater than 10-20TB, how does using all SSD's work or even become affordable?

Share this post


Link to post
5 hours ago, busa1 said:

Any update on this? Almost 3 years later, an all SSD array is more realistic than ever. Can we expect unraid to support this anytime soon officially? 

Don't expect array support soon, but you can already have up to a 24 device cache pool, and btrfs raid5 is mostly OK now, I've been using it for some time now, but still do frequent backups of course.

Share this post


Link to post
9 hours ago, ashman70 said:

Not sure how you come to the conclusion that an all SSD is more realistic, for who? SSD's beyond 2TB are quite expensive and if I want a storage array greater than 10-20TB, how does using all SSD's work or even become affordable?

A 4TB QVO SSD costs around 100 euro/TB which is about the cost of a brand new SAS drive also. And if you are looking at 2.5 form factor, hard drives are getting pretty expensive if you are planning to go for high density on a small form factor. I do not even know if there are any 2.5" drives that support 4TB... So SSDs are really breaking into the market now. I'm looking for some drives for a R620, and thus I cannot go the 3.5" form factor, but still want large capacity, then I'll have to go with SSDs.

 

4 hours ago, johnnie.black said:

Don't expect array support soon, but you can already have up to a 24 device cache pool, and btrfs raid5 is mostly OK now, I've been using it for some time now, but still do frequent backups of course.

Can you do dynamic updates to this pool? Could you explain exactly how your setup works? I'm planning to start out with only 2-3 disks, but wanna go up to 10 disks. Is it possible to dynamically grow this pool without having much downtime? I do not want to have a full backup of the data that I'll be running on this server. All I need is a single disk redundancy, that is why I looked into UnRaid for this solution. 

Share this post


Link to post
3 minutes ago, busa1 said:

Can you do dynamic updates to this pool?

Yes, btrfs supports online disk replacement, also adding and removing disks, you can even do them without stopping the array, though normally it's done using the cache pool functions, similar to array disk replacements/upgrades, I currently have an 11 SSD raid5 cache pool in one of my servers.

 

8 minutes ago, busa1 said:

A 4TB QVO SSD costs around 100 euro/TB which is about the cost of a brand new SAS drive also

Yes, but keep in mind that write speed in these QLC SSDs after the small SLC cache if filled up downgrades to 160MB/s, so not much faster that a normal disk for sequential writes, though it will be for reads or random access.

 

Share this post


Link to post
12 minutes ago, johnnie.black said:

Yes, btrfs supports online disk replacement, also adding and removing disks, you can even do them without stopping the array, though normally it's done using the cache pool functions, similar to array disk replacements/upgrades, I currently have an 11 SSD raid5 cache pool in one of my servers.

You keep mentioning that you run it in raid5 cache pool. Is this something that unraid offers? Or you run a dedicated raid controller and place all your disks in to raid5, and then pass that as an individual disk to unraid? Could you please describe what kind of setup you have? Also I haven't been experimenting with the cache pool functions that much, so I'm not really sure what options are there. 

 

14 minutes ago, johnnie.black said:

Yes, but keep in mind that write speed in these QLC SSDs after the small SLC cache if filled up downgrades to 160MB/s, so not much faster that a normal disk for sequential writes, though it will be for reads or random access.

Yes, I'm fully aware of this, as I will be running multiple VMs there will be hardly any read writes to the disks (only temporary during a cloning procedure), maximizing random read and writes would be my primary goal, so the VMs would have better responsiveness. 

Share this post


Link to post
3 minutes ago, busa1 said:

You keep mentioning that you run it in raid5 cache pool. Is this something that unraid offers?

By default it's raid1, but it can be changed:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=480421

 

You can also have multiple pools using UD with some limitations:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=462135

 

Share this post


Link to post
17 minutes ago, johnnie.black said:

This raid5 with BTRFS sounds exactly what I'm looking for. I'm gonna use Esxi and put unraid on top of it, and share back an NFS share to ESXi for the VMs. When I run out of space, just add another disk, and dynamically increase the possible NFS share. One more question: if a disk fails does it has to be the same size or larger than the disk it will be replacing? 

Share this post


Link to post
23 minutes ago, busa1 said:

One more question: if a disk fails does it has to be the same size or larger than the disk it will be replacing? 

To be easily replaceable yes, though btrfs allows it to be smaller if the filesystem is manually resized before replacing.

Share this post


Link to post

I'm gonna share my all-ssd tryouts...

In total 6 ssd 860 evo 1TB + intel s1200splr + 32 GB ECC + onboard sata (AHCI)...
During the creation of the array and initialization of the 2 disks that were allocated under parity reported about CRC error (SMART). The support team was very surprised that this had happened, but said that the SSD was still "at your own risk". 

Share this post


Link to post

CRC errors are typically bad connections. What support team? I doubt anyone here would be surprised by CRC error.

 

Might be instructive for you to read the rest of this thread to better understand why SSDs aren't recommended in the parity array.

Share this post


Link to post
18 minutes ago, trurl said:

CRC errors are typically bad connections.

 

No, there's no problem with the cable. These same disks were built in ZFS and there are no problems there. Sata cable using SuperMicro CBL-0044L

 

20 minutes ago, trurl said:

What support team? I doubt anyone here would be surprised by CRC error.

I wrote to your support team and got this answer: "The CRC issue is odd, but I do suspect your issues being related to how discard/trim works differently on various SSDs.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.