Is there a way to attach external case with just hotswap bays and HDDs to workstation running Unraid?


JKunraid

Recommended Posts

I am in the process of building my first Unraid workstation/server! 

 

The use case is a mix of home lab (assortment of docker images), Plex media server, backup server, GPU miner, ploter and storage for Chia, Steam streaming, and eventually some home automation software (using docker). To keep costs down I'm building it out of a mostly repurposed workstation I already own rather than go out and buy all new hardware specifically for a server build.

 

 

My current workstation case only holds up to 12 HDDs, but my understanding is Unraid supports up to 30 HDDs on single volume (plan to use dual parity) To bring down costs a bit I'd prefer if I could add more HDDs to some external hotswap bay "server" but without any server parts in it (i.e. still using Unraid on my workstation/server as opposed to separate hardware).  The sole purpose of the secondary external case would be to hold Hotswap bays for additional drives not to act as a second server, so from the appearance of Unraid it should seem like all the equipment is in the same case.   Is this possible to do with Unraid and if so what hardware would I specifically need to do it? (e.g. is there any inexpensive 12, 24 or 36 bay hotswap cases designed for such a use case.  What sort of HBA adapter would I use to connect the hard drives in the two cases? etc)

 

And as (a more expensive) alternative, is there an inexpensive case someone could recommend that has 32-36 hotswap bays that I could use below parts with? (j.e. just changing my case. HBA card) 

 

-----------------------------

Case: Phantiks Enthoo 719 (supports upto x12 3."5 drives, along with remaining space for x3 2.5" drives)

Mobo: Asus Zeneith II Extreme   (4 PCIe 4.0 slots, USB 3.2 support, 10Gbe rj-45, x5 M.2 NVMe slots. upto 8 sata drives if 5th NVMe slot disabled otherwise 4)

CPU: Threadripper 3970x

RAM: 64GB non-ECC 3200MHZ (will switch to 256GB ECC in the future)  

Evga RTX 3090 FTW3 (possibly second basic card if I can't figure out way to do shared passthrough vGPU with it)

LSI 9240-8i Sata/SAS HBA Adapter  (might switch to a 16 or 24 port)

ASUS Hypercard  (v.2 PCIe 4.0)

x3 Intel p4610 SSDS (attached via U.2 to M.2 adapter to ASUS Hypercard)

x5 2TB PCIe 4.0 NVMe SSDS (4 attached to the motherboard, one to the ASUS hypercard)

Cooling: Mix of Noctua CPU, 140 and 120mm case fans 

1600W Platinum ATX PSU

HDDs: all 18TB (total number of drives TBD)

 

Rack: Holds both standard 19" and has shelves for non-rackmount equipment 

 

 

 

 

Edited by JKunraid
forgot to add psu
Link to comment
31 minutes ago, ChatNoir said:

Just to be clear, Unraid is not plug and play for Array and Pools drives.

 

Hotswap bays are convenient for many reasons, but not directly usable in Unraid.

 

It didn't even occur to me that I couldn't just hotswap a failed/failing drive and then tell to Unraid to add the new drive to the array. Are you saying I need to power down the machine if a drive fails? Can you elaborate what you mean by "not directly usuable in Unraid".

Link to comment

The array needs to go offline to replace disks.

While the array is offline, this will also shutdown all VM's and docker containers.

Hotswap is your friend at this point as it will let you replaced failed drives without needing to mess with actual connections. Which many users in the past have managed to disconnect additional drives while the case is open.

Hotswap will not require a shutdown, but just about everything else built in, will be stopped. and resumed only once the array is back online.

 

Once the array is offline, any drive of the array can be replaced (including a parity drive), with Unraid prompting to start a rebuild of the drive based on the state of the array, once the array has been restarted.

 

Link to comment
7 hours ago, jonathanm said:

 

 

Thanks for link. I have a couple of 4u case laying around so something like this might work. It's opened up a proverbial rabbit hole of more questions though.

 

1. The poster says he used an HBA card with external ports  in primary case (the actual server) which uses some cable to connect to a "SFF-8088 to 8087 adapter" in secondary case that in turn connects to 24 port RES2SV240 expander (powered by molex) which in turn has breakout cables to power the drives in secondary case. Is that a correct synopsis?

 

Assuming the above is true....

 

2.  Would this HBA card work?

 

 

3.  What is the name of the cable he used? (assuming SFF-808 but just to confirm)

https://www.amazon.ca/Deconn-External-SFF-8088-Cable-Attached/dp/B00S7KTXW6/ref=sr_1_1_sspa?dchild=1&keywords=sff-8088&qid=1623055881&sr=8-1-spons&psc=1&smid=A3CMOOTCHB9X54&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExVjBDRVY5STM4RkRYJmVuY3J5cHRlZElkPUEwOTI5ODk2M1FYSTc2SEJDREM1NyZlbmNyeXB0ZWRBZElkPUEwNTM5OTgyMTRKQ1BKWldDV0g2TyZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=

 

4. What SFF-8088 to 8087 adapter would you recommend that compatible with Unraid?

 

5. What would power the above SFF-8088 to 8087 adapter? Is there some cable I could use to connect to main case? 

 

6. On similar vein, what is powering the hdds in secondary case? Is it being powered from sata power from first case?

  

 

 

 

 

 

Edited by JKunraid
additional details
Link to comment

5

57 minutes ago, ken-ji said:

The array needs to go offline to replace disks.

While the array is offline, this will also shutdown all VM's and docker containers.

Hotswap is your friend at this point as it will let you replaced failed drives without needing to mess with actual connections. Which many users in the past have managed to disconnect additional drives while the case is open.

Hotswap will not require a shutdown, but just about everything else built in, will be stopped. and resumed only once the array is back online.

 

Once the array is offline, any drive of the array can be replaced (including a parity drive), with Unraid prompting to start a rebuild of the drive based on the state of the array, once the array has been restarted.

 

 

"The array needs to go offline to replace disks."

 

 

Argh. For clarification, which option below are you describing? (assuming option A but just to confirm)

 

A.  array would remain up (in degraded mode) even with bad disk (assuming parity disk(s) exists). if I wanted to fix the issue though, I would need to manually shut down array to physically add new drive (shutting down any programs that depend on it), then manually add new drive to Unraid array, --- and then array would be active while rebuilding? (i.e just a few minutes of downtime)

 

B. array would immediately go down while in degraded mode. (i.e. could take awhile before I notice)

 

C array would remain up in degraded mode and would stay down during rebuild?  (i.e. rebuild would take the system down for days if using 18GB drives)

 

D. array would be down while degraded and would stay down during rebuild? (i.e. rebuild would take the system down for days if using 18GB drive)

 

E. Other?

 

Secondary related question...

 

The bulk of the data is non-critical storage (e..g Chia plots, multi-media, backups). is there a way to create a couple of additional smaller arrays (e.g using M.2 drives) specifically for the purpose of mirroring critical storage so if one array goes down the other array can keep critical apps up while I work on restoring the array with the problem? (e.g. docker images running development webserver, bitwarden, bookmark sync software, etc)  

 

 

Link to comment

A. when a disk goes bad, the parity protection of the array - like RAID 5 (or 6 if you have two parity discs) - allow the array to continue functioning in degraded mode. If you have hotswap, to replace the disk(s) you will need to stop the array, which will disable anything using it including the docker engine and VM engine. After replacing the disk(s), the array can be started in the same degraded mode, and Unraid will rebuild the replacement disk or recompute parity - depending on which disk(s) gone bad. Your total downtime is just the time to stop the array, replace disc, and start it up again.

 

2nd A.

Storage in Unraid is basically in three tiers: Array (usually parity protected and support shares that span the array), Cache (these are independent pools of drives if using btrfs, for either caching writes to the array or keeping data on faster disks) and Unassigned (this requires a plugin to simplify, but are storage not in previous tiers and mounted in whereever you want)

There is unfortunately only one Array per server, so most users backup to either unassigned disks that they periodically mount, backup, unmount or they target another remote server and use whatever backup tool (ie rsync, google drive, backblaze b2, etc)

 

and as I mentioned, docker containers and VMs are tied to array running, so stopping the array stops everything, but with hotswap bays you can be up and running in a few minutes it takes to stop the array and replace the drives.

Link to comment
1 hour ago, JKunraid said:

The poster says he used an HBA card with external ports  in primary case (the actual server) which uses some cable to connect to a "SFF-8088 to 8087 adapter" in secondary case that in turn connects to 24 port RES2SV240 expander (powered by molex) which in turn has breakout cables to power the drives in secondary case. Is that a correct synopsis?

Yes.

 

1 hour ago, JKunraid said:

Would this HBA card work?

Which one?

 

1 hour ago, JKunraid said:

What is the name of the cable he used? (assuming SFF-808 but just to confirm)

That's it, SFF-8088 to SFF-8088.

 

1 hour ago, JKunraid said:

What SFF-8088 to 8087 adapter would you recommend that compatible with Unraid?

Any one will work, they are transparent to the OS.

 

1 hour ago, JKunraid said:

What would power the above SFF-8088 to 8087 adapter? Is there some cable I could use to connect to main case?

It's not powered, looks like this:

image.png.8bc779f5f27820575629719cc0e08cbc.png

1 hour ago, JKunraid said:

On similar vein, what is powering the hdds in secondary case? Is it being powered from sata power from first case?

In that case I had a second PSU to power the disks and the expander on the second chassis.

Link to comment
5 hours ago, JKunraid said:

I have a couple of 4u case laying around so something like this might work.

 

In this post you will find typical hardware that can be used to bind several external chassis to one main server. It's one way to do things like these, there are different others available depending on your needings.

 

 

 

Link to comment
4 hours ago, JorgeB said:

Which one? (HBA)

 

 

I was going to provide a link to dedicated external HBA card but just realized I have issue with it.

 

I currently have an in  8-port LSI  HBA (internal only) and was planning to use the spare SATA ports on the mobo for the other four bays case supports.  As I only have four PCIe ports to work with (other three have been allocated elsewhere) could I replace my LSI HBA with another 8 port HBA that also has two 2 external SFF-8088 ports?  (which would then connect to other case's 8088-to-8087 adapter, which in turn would connect to the Intel expander you mention right).

 

If so, as this is a budget build, could you recommend an HBA from ebay? (usually can find less expensive imports from Asia or used ones)

 

 

 

 

Link to comment
23 minutes ago, hawihoney said:

 

In this post you will find typical hardware that can be used to bind several external chassis to one main server. It's one way to do things like these, there are different others available depending on your needings.

 

 

 

 

Thanks for link. Fun read.

 

Ideally I would buy a case with 30+ accessible bays, mobo, cpu, etc...if starting from scratch. At the moment I'm trying to save money on by repurposing my existing equipment as much as possible.

Link to comment
37 minutes ago, hawihoney said:

 

In this post you will find typical hardware that can be used to bind several external chassis to one main server. It's one way to do things like these, there are different others available depending on your needings.

 

 

 

 

Btw -  Someone told me Unraid supports up to 30 drives in array but only 1 array.. is that 30 drives total in Unraid  (i.e. 28 usuable + 2 parity)  or could have additional unassigned drives past 30?

 

For instance, although a less than optimal solution it could still be useful to have a local backup of critical data broken up to various unassigned drives, 

 

Or say some extra hotswap bays where I could  import  data manually from a drive from another machine or do a drive to drive copy. (does unraid allow me to do that?)

Edited by JKunraid
typos
Link to comment

The thing about hotswap is that even if the hardware allows it, Unraid won't actually use it to initiate anything. You don't have to power down, but you do have to stop the array.

 

Before Unraid will do anything at all with a new disk, you have to assign it. If it is a replacement disk, it has to be assigned to the same slot as the disk it is replacing. If it is intended to be an added disk, it has to be assigned to a new slot.

 

There is no way to make any disk assignments or changes while the array is started. Dockers and VMs don't run, and data is inaccessible while the array is stopped.

 

Once you start the array again, if Unraid sees a replacement disk has been assigned, it will begin rebuild. If it sees a new disk has been added, it will clear it (unless precleared) so parity remains valid, but the new disk isn't actually added and so can't be used until clearing has finished.

 

Degraded array still works in every way with docker, VMs, data accessible, but there may be some performance impact due to emulation of disabled disks or to rebuilding.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.