HYEPYC | High End EPYC Milan Build


Recommended Posts

Bit of an update on the build as I've had it for almost two months now.

 

HARD DRIVE FAILURE & REPLACEMENT

 

One of the 10TB drives received 20,000 errors during the first parity read check. This was the last drive inserted into the system and hadn't yet received any data. This wasn't too unexpected due to it being 3.9 years old. Anyway I promptly removed it and replaced it with another 18TB drive from a WD MyBook, the drive inside was identical to the WD Elements I already shucked which was good.

 

So the total setup now is 5 x 18TB and 6 x 10TB with dual-parity enabled. Right now it's filling nicely, I've gone from 43TB used to 48.1TB used already and that is continuing to grow, with some future plans I think it'll be near to 90TB used by January. I have decided to switch out the 10TB drives with 18TB ones before adding more drives to the array due to them getting old and one has now partially failed.

 

I did decide to keep an 18TB drive pre-cleared on a shelf. One I've tested rigorously and know can be used as a replacement when I need it. So far the server is running wonderfully, extremely quiet still, can't tell it's on without the blinking lights, great cooling with all the drives remaining around 31-33c and the CPU not going much over 50c. Power consumption is also good, about 175 Watts 99% of the time with brief increases to 210-220 Watts during an array parity check.

 

GENERAL STABILITY AFTER A MONTH

 

I've not had any lockups or crashes so this hardware seems well at home running unRAID.

 

SMB SHARE PERFORMANCE WOES WITH MACOS

 

The only active problem I'm having with unRAID since the very beginning is its very poor macOS compatibility. The performance and stability of SMB shares from unRAID under the latest macOS (Monterey) are abysmal. Sometimes I can have days where it works perfectly and other days I'm seeing share disconnections, files taking 10-15 seconds to open, and videos being streamed to a media player pausing every few seconds. It's bad.

 

These issues do not apply to SMB shares on Windows however where it works perfectly. I'm able to hit 5GB/s (40Gb/s) via SMB share to a Windows VM running on unRAID, I can only dream of speeds close to that on my Mac which tops out around 3.75MB/s (30Mb/s). From a lot of googling and discussing this with people on the unRAID discord, it appears to just be how it is and everyone experiences the same issues to differing degrees.

 

This issue is quite annoying and I do hope this situation will be improved in the future as I wasn't aware of it before I started my unRAID journey and it isn't something I experienced with my previous setup which ran Windows Server 2012 R2. SMB shares from that server worked perfectly with macOS.

 

VM NETWORK ISSUES & RESOLUTION

 

One strange issue that I encountered with my VMs was their networking was acting strangely under high load. For instance, if I'm transferring files at near 10Gb/s I would have my VNC or RDP sessions (hosted from inside the VMs themselves) stutter, pause and disconnect. Similarly, any network services inside the VMs would experience the same problems under high network activity.

 

I spent a long time investigating the problem, trying to narrow things down. The issue did go away when passing through a physical 10Gb network card to the VM but I wanted to use the network bridge created by unRAID for all of my VMs since I can't put in so many network cards for each VM that I wanted to operate.

 

Well, it turns out the default "Network Model" selected by unRAID is called virtio-net and the way this works presents a lot of overhead. A virtualized NIC is used to handle packets in and out of the VM inefficiently when compared to the virtio (without the -net) method which interacts with the VM in host memory making it much more efficient.

 

So essentially by switching from virtio-net to virtio as my network model I was able to eliminate the network degradation I was experiencing. No more did my VNC or RDP sessions collapse under heavy network use and in fact via iPerf I was able to raise network performance from 9.8Gb/s to 38Gb/s when testing between a Windows VM and the unRAID Host.

 

I should mention there is a big warning on using virtio instead of virtio-net, and that is it may make the system unstable when using the bridge network it creates with both virtual machines and docker containers at the same time. I've yet to experience this but I'm not using Docker much, instead preferring to run things from virtual machines.

 

MY GENERAL OPINION OF UNRAID AND WOULD I DO THIS AGAIN

 

So far I do love unRAID. I have the sense it's very polished and the community has filled in the gaps where needed, plugins like the unassigned devices stuff, pre-clear plugins, and unbalance. All of these shore up the operating system where it's needed and I'm very happy with it. I for sure would build another in the future and I of course intend to keep this one. I don't have any regrets about choosing unRAID over TrueNAS.

 

THINGS I WOULD LIKE TO SEE IMPROVED IN UNRAID

 

I feel it needs a proper bulletproof backup system. Right now backing up Virtual Machines isn't simple, there is a plugin but it's broken and the maintainer is seemingly MIA. I feel this is important enough to include in the base system.

 

The same situation with backing up Docker containers, there is a plugin but it requires your dockers to be powered down to perform the backups. If I'm hosting Plex for instance as a docker and I schedule backups to happen every day then my family is going to get quite annoyed if they're watching something when the backup occurs. Yes, this can be mitigated by performing backups at say 4 am but it can still happen and backups shouldn't be disruptive.

 

I'd also like to see the SMB performance on macOS meet parity with Windows. I don't have these same issues when hosting SMB shares from Windows Server or TrueNAS so I'm confident in saying this is something the unRAID team need to resolve.

 

The second to last thing I'd like to see improved is right now when you have dual parity you cannot move drives around without losing parity on your second drive. I understand this is a limitation based on how the number of drives is used as part of the parity calculation and I don't have a problem with that, but when you use the drive fill mode on a share (fills one drive at a time in sequence) you may have a drive lower down that you want to be filled before higher drives.

 

For instance, I replaced a 10TB drive with an 18TB. But my shiny new 18TB drive is at the bottom of my disks now and so the fill method doesn't yet use this drive and won't do so for almost another 40TB of data. I'd prefer if the OS had a way to assign drives so we can swap them around virtually with a remapping table to their physical allocations. This way I could position all my 18TB drives together in the UI so the share fill method worked as I would prefer it.

 

The last thing I'd like to improve is the boot method. I would appreciate the ability to boot the OS from RAID1 drives. I would 100% install two SATA SSDs in RAID1 just for a more reliable boot method. The USB key thing is convenient but unreliable and so I'd like an additional boot method (keep USB keys, but add normal SATA/NVMe booting with RAID1 capability, please!)

 

So that's about it, not too much I'd change, the OS gets hundreds of things right to a handful of things wrong. This is a great batting average.

Edited by Pri
Link to comment
11 hours ago, Pri said:

Right now backing up Virtual Machines isn't simple, there is a plugin but it's broken and the maintainer is seemingly MIA. I feel this is important enough to include in the base system.

tldr: Use backup software inside the guest OS so you can keep the guest OS running during backup, and restore anywhere, same strategy you would use for a bare metal machine.

 

Counterpoint.

Any backup operation running exclusively on the host is going to require the VM to be shut down for backup so the guest is in a stable state. If you back up a VM while the guest has files open in RAM, what is backed up isn't going to be fully valid.

 

Therefore, I suggest that your guests should be running backups in the guest OS, so the backup agent can utilize shadow copies or whatever is needed to get a good backup while the guest OS is kept running.

 

I agree that Unraid needs to do a better job of allowing us to keep good VM definition backups, it sure would be nice to have XML backups and versioning without using 3rd party plugins. It shouldn't be a catastrophic event if you muff an XML edit, a rollback button would be ideal.

 

11 hours ago, Pri said:

I would appreciate the ability to boot the OS from RAID1 drives.

Not sure what that would gain, and you would lose the use of a couple interfaces that could be used for storage.

 

It may sound like nit picking, but it really isn't, Unraid technically boots and runs from RAM, not the flash drive. The OS files are read from the flash archive and unpacked into RAM, where the boot process really starts. The flash is essentially read only for the purpose of being called "boot media", and read operations don't significantly effect flash lifetime expectancy. Only configuration changes are written to the USB so they can be applied at the next boot.

 

Downloading a flash backup from the management GUI whenever you make significant changes is easy and quick, and if your boot USB fails, it's a quick and easy replacement. I know to you as a relative newcomer to Unraid having the OS dependent on a USB flash drive feels wrong, but it's a proven strategy that works, and allows much better portability and ease of migration to new hardware. Good USB sticks will run Unraid for MANY years, there are plenty of Unraid installations that have moved the same USB stick through several generations of motherboards and sets of hard drives.

 

11 hours ago, Pri said:

my shiny new 18TB drive is at the bottom of my disks now and so the fill method doesn't yet use this drive and won't do so for almost another 40TB of data.

I understand the OCD of wanting to keep things visually neat and in order. I'm pretty sure the vast majority of Unraid users have some level of OCD, some are even to the point of CDO, where even the letters of the syndrome must be in order.

 

However...

If you have that much excess space, I would argue you have too many drives. All drives, even empty ones, are used in their entirety in the parity calculation. So having empty space presents unneeded risk, energy consumption, and drive wear hours.

I follow and recommend the practice of keeping between 1 to 2X the size of your largest drive free, so for your rig I would want less than 36GB total free, and only when that reduced to 18GB free would I add or upgrade a drive.

 

That obviously doesn't apply during initial data load, where you know pretty much exactly how much space you need, but during normal use where data growth is relatively predictable it works for me.

Link to comment

Dear Jonathan,

 

I already use Veeam agents inside my VM's to do backups inside the VM's. What I would like from Unraid is backing up of all the settings surrounding the VM's in the host itself, not of the VDISKS but of the settings and so forth. I'd also like snapshots but that's a separate thing. I'd also like proper Docker backups. I find myself just using a VM instead currently due to the backup situation.

 

Quote

Not sure what that would gain, and you would lose the use of a couple interfaces that could be used for storage. 

It may sound like nit picking, but it really isn't, Unraid technically boots and runs from RAM, not the flash drive. The OS files are read from the flash archive and unpacked into RAM, where the boot process really starts. The flash is essentially read only for the purpose of being called "boot media", and read operations don't significantly effect flash lifetime expectancy. Only configuration changes are written to the USB so they can be applied at the next boot.

 

I mean.. what it would bring is redundancy against failure and a higher chance that the system will boot from cold without issue. That's kind of the point of RAID1, redundancy. I can also run pfSense from memory (it has the option) and yet I still run RAID1 drives there too. I'm fully aware that unRAID runs from memory and that doesn't change my view on this subject.

 

However I'm not suggesting that the USB method should go away, just that we get some additional ways to install and boot unRAID for those of us who want it.

 

Quote

Downloading a flash backup from the management GUI whenever you make significant changes is easy and quick, and if your boot USB fails, it's a quick and easy replacement. I know to you as a relative newcomer to Unraid having the OS dependent on a USB flash drive feels wrong, but it's a proven strategy that works, and allows much better portability and ease of migration to new hardware. Good USB sticks will run Unraid for MANY years, there are plenty of Unraid installations that have moved the same USB stick through several generations of motherboards and sets of hard drives.

 

I do think it feels wrong, yes. I think we will just have to agree to disagree on this subject.

 

Quote

I understand the OCD of wanting to keep things visually neat and in order. I'm pretty sure the vast majority of Unraid users have some level of OCD, some are even to the point of CDO, where even the letters of the syndrome must be in order.

 

However...

If you have that much excess space, I would argue you have too many drives. All drives, even empty ones, are used in their entirety in the parity calculation. So having empty space presents unneeded risk, energy consumption, and drive wear hours.

I follow and recommend the practice of keeping between 1 to 2X the size of your largest drive free, so for your rig I would want less than 36GB total free, and only when that reduced to 18GB free would I add or upgrade a drive.

 

That obviously doesn't apply during initial data load, where you know pretty much exactly how much space you need, but during normal use where data growth is relatively predictable it works for me.

 

It's not OCD that is my rationale, it's that I want my freshest drive that is youngest and will (hopefully) last the longest to be filled before my older drives that I'm in the process of swapping out, I'll likely remove the oldest (and highest) 10TB drive soon to remedy this. I'm adding about 8-12TB a month currently but that will be increasing. I expect to have all my 10TB drives removed by the end of the year and replaced with 18TB models and then the system will also be at near capacity around the same time frame.

Link to comment
  • 3 months later...

I have an update to the build today that may be of interest to others. I've installed 4 x 2TB Samsung 980 Pro's to use as a RAID0 cache (this is only for non-essential data).

 

I came across an issue with my original 2TB cache in that it was too small. On some days I would put over 2TB into it requiring multiple moves in a single day and I also wanted the cache drive to hold its data for a few days to a week so that my software can interact with the data stored there in a faster manner than is possible with the unraid array itself.

 

As a result I got the 4 x 2TB Samsungs and another PCIe card to place them and installed that into the server as shown below.

 

L25m7dxk8.jpg

 

And below is a picture of it installed into the server with my other add-in cards. You may notice there is no GTX 1080 Ti installed (mentioned on page 1) and that's because I don't need it at present and so I decided to remove it from the server as to extend the cards working life.

 

mXdKXARQH.jpg

 

With this addition to the server I decided to transition my previous cache drive (2TB SN850) into a 2nd drive for my VM's and Dockers and so I put both of them in RAID1. This was quite painless to do. Now I'm sure you're all curious about the performance of 4 x 2TB 980 Pro in RAID0.. as was I.. but there is an issue with the current 5.19 Linux Kernel when combined with EPYC CPU's which introduces a dummy I/O wait and that significantly curbs performance.

 

This is a regression that wasn't present in earlier kernels and is already fixed in 6.x kernels and because of that I'm restricted to 2.1GB/s and this has been confirmed by other users that they are facing the same hard limits with the same kernel combined with EPYC.

 

And that brings me to the performance screenshot below:

 

Rm9BRjpjX.jpg

 

Once the kernel issues are resolved I'll post an updated screenshot of how this really performs. Until then I hope this was at-least interesting. I really like these simple PCIe to M.2 adapter cards (which require bifurcation on your motherboard). The one I purchased 7 months ago is still working flawlessly and if you are buying M.2 drives which already have heatsinks as I am you can see in the above screenshot that the temperatures can be really good.

 

As of now the server has 14TB of NVMe storage (1x2TB 970 Evo Plus, 2x2TB SN850, 4x2TB 980 Pro), quite a lot more than I initially thought I would use, I also have 10 x M.2 slots including two on the motherboard (1 unused) and 4 on each PCIe card (2 slots unused).

  • Like 1
Link to comment

Very nice setup. I'm looking into a similar setup... a few gens older than yours mind you but its great to see what will be an option. I'm on consumer hard ware atm (3900x) which is absolutely fine but ive just filled up a 24bay server and the lack of pcie lanes is a big issue for me now. So im looking to move to an epyc setup with the same case you are using and then use my 2nd 24bay as a jbod. 

 

Do you think you might go with a zfs cache if/when that becomes natively available? 

  • Like 1
Link to comment
1 minute ago, tazire said:

Very nice setup. I'm looking into a similar setup... a few gens older than yours mind you but its great to see what will be an option. I'm on consumer hard ware atm (3900x) which is absolutely fine but ive just filled up a 24bay server and the lack of pcie lanes is a big issue for me now. So im looking to move to an epyc setup with the same case you are using and then use my 2nd 24bay as a jbod. 

 

Do you think you might go with a zfs cache if/when that becomes natively available? 

 

I do hope you will post your finished build on the forums, they're great to admire!

 

I will definitely test the ZFS pool performance compared to BTRFS once it's officially available and if it's as-fast or faster I'll probably use it instead.

Link to comment

 

1 hour ago, Pri said:

 

I do hope you will post your finished build on the forums, they're great to admire!

 

I will definitely test the ZFS pool performance compared to BTRFS once it's officially available and if it's as-fast or faster I'll probably use it instead.

 

I will but it might take a while. The plan is to do it in stages. Stage 1 case change. Stage 2 change the guts of it to epyc. Stage 3 add the jbod. So god knows how long before I get it finished. The case alone will cost me the guts off 1k.... then another 1-1.5k for stage 2 and then maybe 500 for the jbod addition. I dont have the funds to get it all done right away unfortunately. 

Link to comment
10 minutes ago, tazire said:

I will but it might take a while. The plan is to do it in stages. Stage 1 case change. Stage 2 change the guts of it to epyc. Stage 3 add the jbod. So god knows how long before I get it finished. The case alone will cost me the guts off 1k.... then another 1-1.5k for stage 2 and then maybe 500 for the jbod addition. I dont have the funds to get it all done right away unfortunately. 

 

Mhm, totally understandable. This chassis is really nice I'm sure you'll like it a lot, will make a great base for your server and all the staged upgrades :)

Link to comment
On 1/7/2023 at 4:10 AM, Pri said:

This is a regression that wasn't present in earlier kernels and is already fixed in 6.x kernels and because of that I'm restricted to 2.1GB/s

Once breakthrough happen, it got interesting. I setup 12x 3.5" disk RAID0, it also up to ~2.1GB/s.

Link to comment
  • 1 month later...

Finally getting around to making my upgrades... I've kinda changed my plan for upgrading and just gone for the whole lot.... and a very similar build to yourself.

 

Gooxi 24-bay Chassis with 12Gb SAS3 Expander

EPYC 7402

Arctic Freezer 4U SP3

SM H12SSL-i

256GB 2666 ECC RAM

Onboard SlimSAS 8i for the main case

2x LSI 9201 16e (for JBOD)

Melenox connectx3 10GB SFP+ NIC

Cyberpower PR2200ERT2U 

 

Still waiting on a few parts to arrive... Slim sas to SFF 8654 is the main hold up atm. So havent posted its own thread about it. Just curious about the noctua A12 fans you got were you able to put them into the hotswap casing? or did you just mount them directly to the chassis? not that it makes a whole lot of difference. Having a few issues with the Motherboard too as it happens but hopefully nothing thats going to be a long term problem. 

Link to comment
1 minute ago, tazire said:

Finally getting around to making my upgrades... I've kinda changed my plan for upgrading and just gone for the whole lot.... and a very similar build to yourself.

 

Gooxi 24-bay Chassis with 12Gb SAS3 Expander

EPYC 7402

Arctic Freezer 4U SP3

SM H12SSL-i

256GB 2666 ECC RAM

Onboard SlimSAS 8i for the main case

2x LSI 9201 16e (for JBOD)

Melenox connectx3 10GB SFP+ NIC

Cyberpower PR2200ERT2U 

 

Still waiting on a few parts to arrive... Slim sas to SFF 8654 is the main hold up atm. So havent posted its own thread about it. Just curious about the noctua A12 fans you got were you able to put them into the hotswap casing? or did you just mount them directly to the chassis? not that it makes a whole lot of difference. Having a few issues with the Motherboard too as it happens but hopefully nothing thats going to be a long term problem. 

 

I look forward to seeing your build!

 

To answer your question. I was able to remove the stock fans from those hotswap plastic shells and reuse the included rubber grommets that the cases use with the Noctua fans. So I maintained the hotswap nature in that I can slide the fans out like normal.

 

However connecting the fans electrically I had to lose that. To do that I simply unscrewed the receiver side which is bolted to the case and fed a PWM cable through. The A12x25 Black's that I have come with a modular 30cm long PWM cable which is sleeved. The fans themselves only have a very short (2cm or so) PWM connector on them.

 

So basically I can pull the fans up, unplug them from their extension cables and have essentially jerry-rigged my own hotswap system in this manner etc - Keep in mind only the black A12x25's have this very short PWM connector on the fans themselves. The brown fans have a normal length cable on them.

Link to comment
7 minutes ago, Pri said:

 

I look forward to seeing your build!

 

To answer your question. I was able to remove the stock fans from those hotswap plastic shells and reuse the included rubber grommets that the cases use with the Noctua fans. So I maintained the hotswap nature in that I can slide the fans out like normal.

 

However connecting the fans electrically I had to lose that. To do that I simply unscrewed the receiver side which is bolted to the case and fed a PWM cable through. The A12x25 Black's that I have come with a modular 30cm long PWM cable which is sleeved. The fans themselves only have a very short (2cm or so) PWM connector on them.

 

So basically I can pull the fans up, unplug them from their extension cables and have essentially jerry-rigged my own hotswap system in this manner etc - Keep in mind only the black A12x25's have this very short PWM connector on the fans themselves. The brown fans have a normal length cable on them.

cheers. Thats what ill do then. 

 

How are you finding the IPMI on that motherboard? I have been having a couple of little niggles with it. In that the temp sensors are showing active but give no information. It also doesnt show my fan speeds. And the fan control doesnt seem to have any effect. Ofc that could be down to the fans rather than the board. 

Link to comment

I've not had any issues with the SuperMicro IPMI features. The fan speeds are correct and all temperature sensors are showing correctly within the IPMI web interface provided by the BMC. I assume you've updated the firmware and BIOS to latest, if not I'd do that.

 

To control my fans precisely I am not using the SuperMicro Fan controls found within their IPMI interface and instead I am using the IPMI Tools package found within the unRAID community apps repository, specifically the one from dmacias.

 

To use that properly you need to set the fans in the SuperMicro IPMI interface to "Full Speed" as this makes the board only send a speed signal to the fans on boot and after that it wont send any more commands allowing the unRAID IPMI software to send its own speed commands without interference.

Link to comment

You were spot on. IPMI and BIOS update solved all the issues I was having. I'm just waiting on a slimsas to 2xminisas cable to arrive to finally deploy the thing. Made the mistake of ordering it on ebay and now its been stuck in NL for the last 10 days waiting to be forwarded on! I ordered a replacement from amazon which should hopefully arrive on time!

  • Upvote 1
Link to comment
22 minutes ago, j44dge said:

Hi there,

 

I'm looking to do a similar build, but I couldn't see what case you're using. Thanks!

 

Howdy, all the specs including the case are link on the first post of the build. The case is a Gooxi 24-bay Chassis. Here is a link to the store I bought it from: https://www.xcase.co.uk/products/gooxi-rmc4124-670-hse-no-psu-atx-psu-barcket

 

The full model number is: RMC4124-670-HSE

Link to comment
  • 2 weeks later...

Nice build! I've put together something similar, maybe I'll throw up a post. My approach is running Unraid as a guest on Proxmox and so far I'm happy. Here's a link to the post on reddit Link with pictures.

 

1x Case SilverStone RM22-312

1x PSU SilverStone GM800-2UG

1x MB Supermicro H12SSL-NT

1x CPU AMD 7443p

1x Heatsink Dynatron A38

8x RAM Kingston KSM32RD4/64HCR 64G (512 G)

1x PCIe U.2 Delock 90077 4x4 v4.0 bifurcation

2x PCIe M.2 Delock 90090 4x4 v4.0 bifurcation

1x LAN Intel X520-DA2 10Gbit SFP+

3x Bracket 2.5" Delock 18271

4x Heatsink M.2 be quiet MC1 Pro

2x Cable Supermicro CBL-SAST-0826

2x Nvme M.2 Samsung PM9A3 1.9 T (Proxmox boot mirror)

2x Nvme U.2 Samsung PM9A3 3.8 T (Proxmox vm mirror)

2x Nvme M.2 Samsung 990 Pro 2 T (TrueNas zfs)

2x Nvme M.2 Samsung 998 Pro 1 T 

2x Nvme M.2 Samsung 960 Evo 1 T (Unraid cache)

2x Optane Intel P4801x 100 G (initially planned as slog for zfs, but decided to run TrueNas nvme only)

1x HDD WD Ultrastar DC H560 20 T (Unraid parity)

4x HDD D Ultrastar DC H550 18 T (72 T) (Unraid pool)

Edited by cbapel
Link to comment
5 hours ago, cbapel said:

Nice build! I've put together something similar, maybe I'll throw up a post. My approach is running Unraid as a guest on Proxmox and so far I'm happy. Here's a link to the post on reddit Link with pictures.

 

1x Case SilverStone RM22-312

1x PSU SilverStone GM800-2UG

1x MB Supermicro H12SSL-NT

1x CPU AMD 7443p

1x Heatsink Dynatron A38

8x RAM Kingston KSM32RD4/64HCR 64G (512 G)

1x PCIe U.2 Delock 900774 x4 v4.0 bifurcation

2x PCIe M.2 Delock 900904 x4 v4.0 bifurcation

1x LAN Intel X520-DA2 10Gbit SFP+

3x Bracket 2.5" Delock 18271

4x Heatsink M.2 be quiet MC1 Pro

2x CableSupermicroCBL-SAST-0826

2x Nvme M.2 Samsung PM9A3 1.9 T (Proxmox boot mirror)

2x Nvme U.2 Samsung PM9A3 3.8 T (Proxmox vm mirror)

2x Nvme M.2 Samsung 990 Pro 2 T (TrueNas zfs)

2x Nvme M.2 Samsung 998 Pro 1 T 

2x Nvme M.2 Samsung 960 Evo 1 T (Unraid cache)

2x Optane Intel P4801x 100 G (initially planned as slog for zfs, but decided to run TrueNas nvme only)

1x HDD WD Ultrastar DC H560 20 T (Unraid parity)

4x HDD D Ultrastar DC H550 18 T (72 T) (Unraid pool)

 

Looks awesome cbapel! - What is the noise and power consumption like?

Link to comment
On 3/9/2023 at 4:55 AM, Pri said:

 

Looks awesome cbapel! - What is the noise and power consumption like?

 

At idle around 160~170 watts and noise with the Standard fan profile is tolerable, keeping the hdd comfortably under 40 C and the flash below 60 C. Though, noise is less of a problem for me because I built a noise and dustproof case around my rack Link .

My goal was never to have to worry about hdd storage capacity and Unraid delivers. At my fill rate and room to grow to 10x 18 T (w 2x for parity) drives I think I've got that covered unless I start running a media focused business.

The tricky part was getting space for U.2 drives. I want everything software related (boot, vm, containers, etc) and zfs based storage to run on flash only. For zfs, this means I get all the nice features without the headache of performance turning and degradation as it fills up. 

Link to comment

Thanks for all the detailed responses, I've already learned a few things here. I like the ipmi workaround you found, and specifics about how Supermicro sets the speed at boot. I am also tempted to HA my OpnSense box as a result (do you just move the WAN cable when trouble starts?).

 

I have a question on how you manage the hba if you don't mind. Right now I have the drives connected to the motherboard controller and I'm passing the devices (hdd) to Unraid as virtio, but I'm on the fence about passing through the hba controller to give Unraid direct access (pros and cons to each approach). Long story short, passing through the onboard AMD hba controller with Proxmox is tricky and I don't like the workarounds that can address the compatibility challenges.

I'm thinking of getting the Broadcom hba you are using, but I'm unsure of how I will get it to work with the three Mini-SAS ports on my backpane. I noticed that your backpane also has three ports, how did you get around this problem? My current setup needs two cables CBL-SAST-0826 from the onboard Slimline-SAS (SFF-8654) x8 ports to the three Mini-SAS HD (SFF-8643) x4 on the backpane. If I went with the Broadcom hba, I would need a cable that splits the x8 into 4 x2, but I'm unaware such a cable exists, or whether Mini-SAS even comes in x2 flavors. Broadcom does have a x16 adapter eHBA 9620-16i from which I can run two cables, but it's around 1'000 USD and hard to find. Maybe I'm missing something obvious, but my first experience with SFF compatibility on an ASRock motherboard has made me paranoid and prone to over complicating. 

 

Cheers

Edited by cbapel
Link to comment
8 hours ago, cbapel said:

I am also tempted to HA my OpnSense box as a result (do you just move the WAN cable when trouble starts?).

  

Essentially yes. I've recently switched my setup up, I put in an Intel X550-T2 (1Gb/2.5Gb/5Gb/10Gb capable) instead of the X540-T2 (1Gb/10Gb capable) into my unRAID box and I've hooked up a single cable from that card to my Modem for WAN (2.5Gb/s). For the LAN side, I'm using a network bridge created by my unRAID box and I pass through a VirtIO nic to my pfSense virtual machine.

 

If I need my dedicated pfSense box for some reason I can just plug the WAN cable from that to my modem and turn it on. I left its LAN cable hooked up to my switch even though its powered off etc

 

I've found pfSense runs really well virtualised on this server and the performance exceeds my dedicated box (I have 1.2Gb home internet). And the energy usage is a lot lower than when I had a physical box. I'm saving almost 50 watts idle by virtualising pfSense as a result.

 

8 hours ago, cbapel said:

Thanks for all the detailed responses, I've already learned a few things here. I like the ipmi workaround you found, and specifics about how Supermicro sets the speed at boot. I am also tempted to HA my OpnSense box as a result (do you just move the WAN cable when trouble starts?).

 

I have a question on how you manage the hba if you don't mind. Right now I have the drives connected to the motherboard controller and I'm passing the devices (hdd) to Unraid as virtio, but I'm on the fence about passing through the hba controller to give Unraid direct access (pros and cons to each approach). Long story short, passing through the onboard AMD hba controller with Proxmox is tricky and I don't like the workarounds that can address the compatibility challenges.

I'm thinking of getting the Broadcom hba you are using, but I'm unsure of how I will get it to work with the three Mini-SAS ports on my backpane. I noticed that your backpane also has three ports, how did you get around this problem? My current setup needs two cables CBL-SAST-0826 from the onboard Slimline-SAS (SFF-8654) x8 ports to the three Mini-SAS HD (SFF-8643) x4 on the backpane. If I went with the Broadcom hba, I would need a cable that splits the x8 into 4 x2, but I'm unaware such a cable exists, or whether Mini-SAS even comes in x2 flavors. Broadcom does have a x16 adapter eHBA 9620-16i from which I can run two cables, but it's around 1'000 USD and hard to find. Maybe I'm missing something obvious, but my first experience with SFF compatibility on an ASRock motherboard has made me paranoid and prone to over complicating. 

 

Cheers

 

Hey cbapel, I'm glad you found the thread useful!

 

So I can answer your questions. Firstly, I have a friend who is passing their HBA through to unRAID from TrueNAS where he runs unRAID in a VM. He's using the same motherboard that you and I are using except he has the SKU with the built in HBA and that is the one he's passing through.

 

For him it works 100%, so we could totally pass through a 9300, 9400, 9500 etc to unRAID from proxmox and that would function properly.

 

As for the SlimSAS cable. It comes with 2 x MiniSAS on one side and 1 x SlimSAS on the other. The SlimSAS side goes into the HBA. So basically the SlimSAS is a PCIe x8 or 8 x SATA/SAS. And the backplane on my case is an expander backplane. It only needs a single cable connection to enable all 24 drives but it has two inwards connections to increase bandwidth. The 3rd connection on my backplane is an output intended to allow you to connect this backplane to another backplane for daisy-chaining.

 

So in my case I have a single 24 slot Backplane at the front. But if I had a 36 bay case which is a 24 bay at the front and 12 bays at the rear a cable could be used to connect those two backplanes enabling all 36 drives to be used from a single or dual-port HBA. In my case the HBA I'm using has a single port that acts like dual-ports when connecting to MiniSAS backplanes due to the breakout cable (SlimSAS to 2xMiniSAS) that is purchasable.

 

The cable I went with was from Broadcom directly. If your backplane is not an expander type then you would need to purchase a HBA that can provide at-least three MiniSAS for your three columns of drives. The 16i models (9300, 9400, 9500, 9600 etc) are popular choices for this.

 

If you do decide to go with a 9500 or 9600 HBA that uses the new SlimSAS connector, please make sure you get the right cable as there are two different varieties, one for PCIe backplanes only and one for SATA/SAS backplanes. If you buy the official Broadcom cables the PCIe only cable is green while the SAS version is black, the two types of cables are not pin compatible even though they'll fit the same receptacles at both ends so it's important to make sure the SlimSAS to MiniSAS cable you purchase is the correct one for your backplane type (I believe in your case you want the pure SAS version).

Edited by Pri
Link to comment
  • 2 weeks later...

I went with 9300-16i in order to get the full bandwidth of the backplane. Like a fool I missed the fact that the H12SSL-i has no on board HBA and waisted time getting slimsas to microsas cables. I've had a lot of headaches with my setup. I have 2 9201-16e to connect to my JBOD... 1 of which now appears to be broken... Which is a pain but 1 still allows me to have 16 of the 24 bays populated. I also got some dodgy 8088 to 8088 wires which was another pain. 

 

At present I am about to fully deploy my server with 2 24bay 4u chassis. 1 of which is fully populated with array drives for ~170TB of storage. I've added 8x1 TB nvme drives to function as 4 separate cache pools. and I have 12 1TB SSD's in the Jbod at present which will be deployed in 2 raidz1 pools. I'm thinking I might wait for Unraid 6.12.0 to deploy these properly. Then I'll add 6 more array data drives as required and finally add an extra vdev of SSD's as required. 

 

My cpu will likely be upgraded at some point. In testing I think the lower clock speeds of the EPYC 7402 along with the 2666Mhz RAM are making the system feel slower in some tasks than the 3900x it replaced. But the benefits of the system far out weigh this minor issue. But I can see myself going for the 7443 as you both have to get the 4Ghz boost. I may look into a RAM upgrade if this doesnt have the desired results. But this is all long term thoughts. For now my server is perfect for my needs. Next on my list will be to look to get my shed setup to accommodate the server. Insulation and wiring still need to be done so that will be the next project.  

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.