Super High Performance Server

[Dr. Ew]

After trying out several boxed NAS solutions, I decided none were as powerful, or as high-performing as I need. 

 

The only solution I could see, was to turn one of my workstations into an open-source NAS. So, over the last several days I’ve tested FreeNAS, Xpenology, OMV, And now UnRAID. After all the testing, I’ve decided, thus far, I like the UI and initial ease of UnRAID the most. I’d like to make this a permanent solution for myself, and several of my employees.

 

I don’t know if I am not understanding, comprehending, or just looking at old information; but, past the main data volumes, I’m a bit confused.

 

A) I’m looking at SSD cache first. I’ve read UnRAID doesn’t support multiple cache pools, yet it appears I have been able to setup multiple pools. But, I’m not sure how it’s working. Unraid let me assign two tri-mode RAID volumes as cache. Volume 1 is 8 256gb NvME’s in RAID5, and Volume 2 is 8 2TB Samsung EVO SSD’s, in RAID 10. It is only showing 3tb of the 8tb SSD RAID Volume, and I’m not sure how much of the NvME volume it’s showing. When I transfer to a share assigned to cache pool, it shows equal in/out on both volumes, but all the data is landing on the SSD volume

 

I don’t understand how this works.

 

B) I have 6 14tb IronWolf Pro’s, attached to the motherboard, which is my main UnRAID data storage. I’m cool with this being my slow storage volume, it’s r/w over 210mb/s and that’s cool for main storage.

 

but, I have a 24-drive RAID 6 array comprised of 8tb Seagate EXOS SAS Drives. Originally UnRAID saw this volume, but it doesn’t show anymore. And, I’m not sure how to setup access to it. The volume is currently empty, but I want to store my warm data on it.

 

the idea is:

Cold Data on the UnRAID data volume 

Warm Data on the External RAID

Hot Data on the NvME-OF And SSD Array

 

How do I achieve this?

 

systems specs 

intel core i9 7280x

ASUS x299 mobo

64gb RAM

6 Internal IronWolf

Intel Tri-Mode RAID x2

Mellanox Connectx-4 or Chelsio t580 NIC

[trurl]

38 minutes ago, Dr. Ew said:

I have 6 14tb IronWolf Pro’s, attached to the motherboard, which is my main UnRAID data storage. I’m cool with this being my slow storage volume, it’s r/w over 210mb/s and that’s cool for main storage.

Assuming those numbers aren't just RAM buffering, this must be without Unraid Parity based on that write speed.

 

40 minutes ago, Dr. Ew said:

I’m looking at SSD cache first. I’ve read UnRAID doesn’t support multiple cache pools, yet it appears I have been able to setup multiple pools. But, I’m not sure how it’s working. Unraid let me assign two tri-mode RAID volumes as cache. Volume 1 is 8 256gb NvME’s in RAID5, and Volume 2 is 8 2TB Samsung EVO SSD’s, in RAID 10. It is only showing 3tb of the 8tb SSD RAID Volume, and I’m not sure how much of the NvME volume it’s showing. When I transfer to a share assigned to cache pool, it shows equal in/out on both volumes, but all the data is landing on the SSD volume

As far as Unraid knows, each of the RAID volumes you are using is a single disk. It is taking those volumes and making a single btrfs cache pool, probably a mirror which is the default.

 

You only talk about storage, which is the original purpose of Unraid, but the way you are using storage, with all those RAID volumes, is very Unraid-like. The normal way to use Unraid is to just give it a bunch of single disks and let it manage them separately with the additional benefit of separate parity disk (or 2) to give some redundancy.

 

And since you don't even mention dockers and VMs (which Unraid has the ability to Host), it kind of makes me wonder how Unraid fits into your picture at all. It has no facility for managing your RAID volumes (except for treating each volume as a single disk), and the RAID volumes are going to make the usual ways we help people with configuring and troubleshooting their systems problematic.

 

Have you explored the Product pages on the website? The Overview in the Wiki might also give you a better idea of what Unraid is.

[Dr. Ew]

1 hour ago, trurl said:

Assuming those numbers aren't just RAM buffering, this must be without Unraid Parity based on that write speed.

Correct, the Parity Drive is still initializing. R/W will suffer decreased performance once the parity drive kicks in? The IronWolf Pro’s hit 200mb/s+ sequential.  

1 hour ago, trurl said:

 

As far as Unraid knows, each of the RAID volumes you are using is a single disk. It is taking those volumes and making a single btrfs cache pool, probably a mirror which is the default.

My initial thought was along those lines, but I don’t see how the math works there. I figured UNRAID would see them as single volumes, but with one volume of ~1.5tb and one of ~8tb, I’m not sure what is mirroring. The available cache space is a strange number. 

 

1 hour ago, trurl said:

You only talk about storage, which is the original purpose of Unraid, but the way you are using storage, with all those RAID volumes, is very Unraid-like. The normal way to use Unraid is to just give it a bunch of single disks and let it manage them separately with the additional benefit of separate parity disk (or 2) to give some redundancy.

The purpose of utilizing UnRAID is multi-fold. Storage utilization is my most critical objective. The 6 IW drives currently dedicated to UnRAID are there as an integral failsafe for my critical, non-replaceable/non-replicable data. I expect UnRAID to be the holding area for my cold, archival, and critical data. The UnRAID data will be replicated on one of the QNAP’s I decided to keep. There will also be a second array (still to-be-determined) that will hold a backup of the critical data. This data currently resides on a DAS Array, on the QNAP, transferring to UnRAID now, and will perhaps occupy another UnRAID Server (the still TBD part).

 

 

1 hour ago, trurl said:

 

And since you don't even mention dockers and VMs (which Unraid has the ability to Host), it kind of makes me wonder how Unraid fits into your picture at all. It has no facility for managing your RAID volumes (except for treating each volume as a single disk), and the RAID volumes are going to make the usual ways we help people with configuring and troubleshooting their systems problematic.

The VM support, and Docker’s will be utilized. It’s also one of the deciding factors, which has sold me on UnRAID, compared to the other options. The VM’s will be spun up to handle A/R content, Research, AI & Deep Learnimg. So, that will be coming, once high-performance is established. 

 

The three RAID array’s exist purely for the needed performance, with protection to keep the data flowing in the even of failure. I don’t need any platform to manage these arrays. The controller management is all that’s needed, given it’s use.

1 hour ago, trurl said:

 

Have you explored the Product pages on the website? The Overview in the Wiki might also give you a better idea of what Unraid is.

I have explored. I’ve read through the documentation and FAQ’s a few times now. But, I’ve also consumed a lot of information on the comparative platforms. With so much information consumption, I undoubtedly have crossed a few concepts into confusion, and I’ve read conflicting information, and some information which still remains unclear, after initial and today’s re-assessment. 

 

After spinning up active installs of the aforementioned platforms, the clear contenders are UnRAID and Windows Server 2019. I would much prefer UnRAID, if I can clear up the high performance needs. Once I decide on the platform, I’m going to purchase licenses for my employees; all who will have similar, but not totally congruent use cases. 

 

My lack of clarity still resides in utilizing cache. The attached cache array is just for testing purposes. Ideally, I would like at least 16tb worth of cache for hot storage. Ideally there will be 8tb of NvME and 8tb of SSD in the cache pool. In addition to the warm additional storage from the 24-Drive Array. I want to be able to offload the cache to the large RAID array and back, with very high transfer rates, and very low waiting time. The 24-drive array, directly attached to a system clears 3gb/s sequential. I need this speed, or close, over the 40gbe network. And I need to eventually expand to 100gbe to cover even higher speed with the NvME-OF.

[trurl]

19 minutes ago, Dr. Ew said:

Correct, the Parity Drive is still initializing. R/W will suffer decreased performance once the parity drive kicks in? The IronWolf Pro’s hit 200mb/s+ sequential.  

By initializing, do you mean you are preclearing it? Or do you mean you are actually building parity? Building parity impacts performance.

 

Reads from the parity array are at the speed of the single disk since Unraid doesn't stripe. Each file is completely contained on a single disk and each disk is an independent filesystem that can be read independently on any Linux. Folders can span disks (user shares). Writes to the parity array will be slower than single disk due to realtime parity updates. See this link for details on how parity is updated:

https://forums.unraid.net/topic/50397-turbo-write/

 

28 minutes ago, Dr. Ew said:

My initial thought was along those lines, but I don’t see how the math works there. I figured UNRAID would see them as single volumes, but with one volume of ~1.5tb and one of ~8tb, I’m not sure what is mirroring. The available cache space is a strange number. 

If you don't want a mirror (and probably not if using different sized "disks" in the pool), other btrfs raid configurations are supported. And you might be better off with some of your volumes as Unassigned Devices instead of in the cache pool. Unassigned Devices are managed by a plugin.

 

See the "Cache Drive/Pool" section of the FAQ for some relevant links:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?page=2&tab=comments#comment-554741

 

 

39 minutes ago, Dr. Ew said:

I’m going to purchase licenses for my employees

Unraid licenses are per server regardless of number of users. Note however that Unraid is not really intended to be a general purpose, multi-user Linux. Only root user has access to the Unraid Management webUI or to the linux command line. All other users only have specified network access to files/folders. Docker/VM access would be controlled by the Docker/VM.

 

Having said all this, I should probably also give a disclaimer. Everyone on this forum (including me) except for those users specifically shown as Administrator are just users of Unraid. Some users do contribute some code through addons etc.

 

But only Administrators work for the company and speak for the company. They can be contacted directly using the "Contact Us" link at the bottom of the page.

 

Free support is through the forum mostly by all of us unpaid and enthusiastic volunteers. Pay support can be negotiated.