First Unraid Setup - Pre-Build Checklist

[SeeGee]

I have never used unraid before, but I have done a fair bit of reading. (Much more to go)

 

Here's the Hardware:

Intel i7 2600K

8GB DDR3 @ 1600Mhz (Non-ECC)

MSI P67A-GD65 Motherboard

• 1 PCIE 2.0 x16 (Nvidia GTX760 SC GPU)
• 1 PCIE 2.0 8x (LSI 9201-8i)
• 4 SATA 6Gb/s ports (SATA1+2 by Intel P67 PCH) (SATA7+8 by Marvell 9128)
• 4 SATA 3Gb/s ports (SATA3-6 by Intel P67 PCH)
• 2 eSATA ports (back panel) by JMicron JMB362
• 8 USB 2.0 ports, 2 USB 3.0 ports(Rear) 2 USB 3.0 ports (Front)
• Realtek RTL8111E Gigabit Ethernet

LSI 9201-8i Controller (2 SAS => 8 SATA 6Gb/s) *NOTE: Awaiting Delivery

 

In total I'll have 12x SATA 6Gb/s, and 4x SATA 3Gb/s available.

There's about a dozen HDD's i'm going to use initially, ranging from 300Gb-2TB.

 

I will be running a couple of virtual machines from time to time (Win10/Ubunt), and Docker. (Yay unRAID!)

 

Here's the Questions:

 

1) Is it better to load the majority of the drives onto the lsi controller, or balance the number of drives on each sata controller? I have a few SATA II drives, and there are 4 SATA II ports on my motherboard, so I'd rather put those drives on the lesser ports. 

 

2) Can I use a raid0 stripe created using the motherboard raid feature as a cache? (I understand this is risky)

 

3) If I cant/shouldn't use a raid0 stripe as a cache, then what should I consider when choosing the size of the SSD I buy to use as cache?

 

 4) I heard about a plugin called "Unassigned Devices". Can I use this to backup the entire contents of the server to external usb drives? YES! Automating this would be helpful. Suggestions on software/docker to use for this?

 

5) Is unRAID more of a single giant file system equaling the total capacity of the data disks (ie: LVM), or is it more like many independent disks stitched together (ie: mounting disks as subdirs). Does it ever span a single file across more than one disk?

 

 

 

 

Edited November 20, 2019 by SeeGee

[trurl]

1 hour ago, SeeGee said:

There's about a dozen HDD's i'm going to use initially, ranging from 300Gb-2TB

I recommend using fewer, larger disks instead. Likely some of those aren't new and maybe not reliable. Fewer disks require fewer ports, less power. Larger disks perform better due to increased density. More disks require more license. Most importantly, more disks are more points of failure. 

[SeeGee]

3 hours ago, trurl said:

I recommend using fewer, larger disks instead. Likely some of those aren't new and maybe not reliable. Fewer disks require fewer ports, less power. Larger disks perform better due to increased density. More disks require more license. Most importantly, more disks are more points of failure. 

Oh, I know. But it's what I have. This mish-mash of disks is what got me looking into unRAID in the first place. these used to be set up in several independent raid arrays. The goal is to make the hardware I have useful, without the nightmare of managing several raid setups. If this setup works out well, I will most certainly be replacing those drives with newer tech when the time/finances allow.

 

On a side note: I found my answer to question 4. Yes, unassigned devices plugin will allow me to backup to external usb drive. (Still looking into the automating it)

 

[JonathanM]

It is imperative that ALL array disks work flawlessly, with their total capacity readable without error. Parity uses the full capacity of all disks, regardless of how full they are, to enable reconstruction of a failed disk.

 

Because of this, I recommend that at the least all used drives must pass a full smart test with no errors. Preferably a preclear pass using either the plugin or the docker.

 

Also, since all drives are fully used for parity regardless of content, I recommend only adding drives as you need capacity. It's not a good idea to add all the drives just because you can, it's much easier to add or upgrade a drive slot than to remove it.

 

There is no worse feeling than losing a drive's worth of data because a couple of unused drives failed while trying to reconstruct the full drive.

[SeeGee]

Good advice. I will make sure to run preclear on them to verify their integrity. Most of these drives have not seen a lot of use. They've spent a few years of their lives sitting on shelves, so their total uptime is relatively low when compared to their age. When you say it's easier to add/upgrade than it is to remove a drive slot, what exactly do you mean? 

 

This is not a mission critical setup. It's a personal use server for home, as well as a bit of an experiment/learning process.

 

 

[trurl]

13 hours ago, SeeGee said:

2) Can I use a raid0 stripe created using the motherboard raid feature as a cache? (I understand this is risky)

Better to just let Unraid manage the btrfs raid cache pool, various configurations are supported:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/#comment-480421

 

13 hours ago, SeeGee said:

3) If I cant/shouldn't use a raid0 stripe as a cache, then what should I consider when choosing the size of the SSD I buy to use as cache?

The total size required for cache depends on how you will use cache. If only using cache for dockers/VMs performance and not caching User Share writes, a large cache isn't needed. This is how I choose to work since my writes are mostly scheduled backups and queued downloads, so I'm not sitting around waiting for them to complete anyway.

 

Others choose to work differently. If you do want to cache writes to User Shares, then you will need larger cache, enough to hold as much as you intend to write between scheduled moves to the array. But keep in mind that you can't move to the array as fast as you can write to cache. Mover is intended to work during idle times.

 

And best to leave cache out of it for the initial data load since cache won't have the capacity to take it all and will just get in the way. Some even leave parity out of the initial data load so they don't have the overhead of realtime parity updates, then build parity later.

[JonathanM]

2 hours ago, SeeGee said:

When you say it's easier to add/upgrade than it is to remove a drive slot, what exactly do you mean? 

Replacing or upgrading a drive slot are built in semi automatic procedures with no loss of parity protection while it's being done if you have 2 parity drives assigned.

 

Adding a new drive slot assignment is automatic with no loss of parity protection under any conditions, as the new drive is filled with zeroes before it is allowed to participate in the parity array, keeping parity valid the entire time.

 

Decommissioning a slot, on the other hand, is a manual process that involves moving any data in that slot either physical or emulated to another location, and either going through the process of writing zeroes to the drive to eliminate it from parity, or rebuilding parity from scratch without the drive.

 

Both the process of moving files and dealing with parity make removing a drive slot a high risk procedure.

 

Normally it's expected that you will populate the array with only the space you need to use currently, and either add slots or upgrade current slots with higher capacity drives as your needs expand. Removing slots isn't a normal everyday thing.

 

Keep your extra drives on the shelf, that way you have replacements.

 

Also... there is no way to directly replace a drive with a smaller drive under any circumstance.

 

If you want some advice on which drives would make the most sense to start the array and which to leave on the shelf, post a list of how many and what size your tested good drives are and how much data capacity is occupied by your current stuff, and we can help formulate a sensible plan for capacity plus expansion and replacements. Make every slot count.

[sota]

One of the downsides of big drives, that I think a lot of people overlook, is the rebuild/recovery time when you drop a disk.  According to my last Parity/Build pass, it took 18 hours, 10 minutes, 19 seconds, Average Speed: 152.9MB/s to process my 1+5 10TB disk array.  That's an awful long time (in my opinion) to be exposed to data loss, and one of the reasons why the next time 10TB drives are on sale the next pair I buy will be used to add the 2nd parity drive and 6th data array. I hoping to buy 3 so I can keep 1 in reserve as a cold spare.

 

[SeeGee]

9 hours ago, trurl said:

Better to just let Unraid manage the btrfs raid cache pool, various configurations are supported:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/#comment-480421

Ok, I see what you mean here. 

• Single Cache Disk = Best to use SSD, (if your cache disk isnt significantly faster than the disks in the data pool, dont bother having one) 
• Dual Cache Disks = Best to use two Identical Drives. Default setup is software RAID1, but you can change that to software RAID0 if you're willing to trade/risk redundancy for performance.

 

9 hours ago, trurl said:

If you do want to cache writes to User Shares, then you will need larger cache, enough to hold as much as you intend to write between scheduled moves to the array. But keep in mind that you can't move to the array as fast as you can write to cache. Mover is intended to work during idle times.

I can run Mover more often than once per day. This is a 2 user network with 6 client machines, and the occasional incoming vpn client used to access files. Almost anytime will be idle time. I do have a 120G SSD that I wanted to put into a laptop, but from what you're saying, it might just be better utilized as a cache drive... I will have to think more on this.

 

10 hours ago, trurl said:

And best to leave cache out of it for the initial data load since cache won't have the capacity to take it all and will just get in the way. Some even leave parity out of the initial data load so they don't have the overhead of realtime parity updates, then build parity later.

This is something I had not thought of! Very useful tip to be honest...

My initial understanding was that it was best to set everything up first: assign cache drive(s), assign parity drive(s), create data pool using every drive you have, test, then start dumping data... While you 'can' do it that way, it's probably more efficient/easier to do it this way.

 

8 hours ago, jonathanm said:

Normally it's expected that you will populate the array with only the space you need to use currently, and either add slots or upgrade current slots with higher capacity drives as your needs expand. Removing slots isn't a normal everyday thing.

 

Keep your extra drives on the shelf, that way you have replacements.

Another great piece of information! It's better to add storage as you need it, rather than creating a massive pool which has to build parity for all the unused space unnecessarily... Makes sense. Can I install the drives into the system but leave them out of the pool as a warm spare? Is it possible to spin down those unused drives?

 

I really appreciate the help you are providing. It is valuable perspective.

[testdasi]

16 hours ago, SeeGee said:

They've spent a few years of their lives sitting on shelves, so their total uptime is relatively low when compared to their age. When you say it's easier to add/upgrade than it is to remove a drive slot, what exactly do you mean? 

Low usage does not equal reliability. In fact, they could be failing and you don't know it simply because they are on the shelves.

 

12 hours ago, sota said:

One of the downsides of big drives, that I think a lot of people overlook, is the rebuild/recovery time when you drop a disk.  According to my last Parity/Build pass, it took 18 hours, 10 minutes, 19 seconds, Average Speed: 152.9MB/s to process my 1+5 10TB disk array.  That's an awful long time (in my opinion) to be exposed to data loss, and one of the reasons why the next time 10TB drives are on sale the next pair I buy will be used to add the 2nd parity drive and 6th data array. I hoping to buy 3 so I can keep 1 in reserve as a cold spare.

I think your downside, while true, is a bit misguided.

• Probability speaking, having 10x5TB almost doubles your probability of having a failure than having 5x10TB, assuming equal probability of failure of 5TB vs 10TB drives.
• Also according to Backblaze data, newer high capacity drives seem to have better reliability than older smaller capacity drives. That means the probability of failure might even more than doubled for the former.
• Then you would have to consider that older drives are usually also slower on average so 10x5TB array will need more than half the time of 5x10TB to build / rebuild. (i.e. the shorter time exposed to risks does not fully offset the probability of such risks).

So yes, building/rebuilding is painfully slow but the first priority should be not needing it to begin with.

 

5 hours ago, SeeGee said:

I can run Mover more often than once per day. This is a 2 user network with 6 client machines, and the occasional incoming vpn client used to access files. Almost anytime will be idle time. I do have a 120G SSD that I wanted to put into a laptop, but from what you're saying, it might just be better utilized as a cache drive... I will have to think more on this.

 

Another great piece of information! It's better to add storage as you need it, rather than creating a massive pool which has to build parity for all the unused space unnecessarily... Makes sense. Can I install the drives into the system but leave them out of the pool as a warm spare? Is it possible to spin down those unused drives?

Ask yourself whether you need the faster speed first before deciding if a share should use cache or not.

• Wifi transfer shouldn't need more than 30MB/s - 50MB/s (within typical "Normal" write speed with parity).
• Wired transfer shouldn't need more than 125MB/s (within typical "Turbo Write" i.e. reconstruct write speed with parity).

The cache functionality was originally created back when HDD was incredibly freaking slow.

Modern recent HDDs are usually fast enough for typical NAS uses that you really don't need cache for such purposes. Cache is more useful for things that actually needs the speed such as appdata, vdisk, temp files etc.

 

You can have install all the drives but leave them out of the array as spares. They should spin down on their own (at least on my current and previous servers).

 

 

[trurl]

On 11/20/2019 at 9:52 PM, SeeGee said:

My initial understanding was that it was best to set everything up first: assign cache drive(s), assign parity drive(s), create data pool using every drive you have, test, then start dumping data... While you 'can' do it that way, it's probably more efficient/easier to do it this way.

But you should install cache before setting up any dockers or VMs, since you don't want to let them get created on the array.

 

On 11/20/2019 at 9:52 PM, SeeGee said:

It's better to add storage as you need it, rather than creating a massive pool which has to build parity for all the unused space unnecessarily...

Building parity is really only about the size of the parity disk. It doesn't matter at all how many data disks there are or how much data is on them. But each data disk must be completely (even the "empty" parts) and reliably read to build parity (or rebuild a data disk, it is basically the same operation).

[SeeGee]

Ok, just a quick update: I have the server up and running! I have a small pool of drives, I also got a cheap 1tb Samsung 860 evo during black Friday, so that is now the cache drive. Performing nicely. Lsi 9211-8i controller is working flawlessly now (had to disable boot support in sas card bios before mobo would boot). I have a Win 10 pro, and Ubuntu VM running smoothly. One major problem though... It seems that the Intel "x" CPUs do support VT-x don't support VT-d, which appears to be necessary for hardware passthrough to the vms... This really broke the value of this project for me... But I have a lead on a replacement cpu for cheap. Crossing my fingers on that... Otherwise things have been going fairly smoothly, and I have learned a lot about unraid. 

 

I'll happily carry on with this thread, but further questions don't fall under the "Pre-Installation" category. 

 

Thanks for your help! It's been useful