Pri

Very cool case, I would have killed for something like this as an alternative to a QNap/Synology/Asustore type 5 bay system when I needed it. I like the low power draw too, thank you for including that in the build information

Great photos! looking forward to seeing more.

haha never mind then. Funny thing is, I even reacted to that post when he made it originally.

Ya know, since you're thinking about changing the "unRAID array" to be a "Primary Pool" perhaps now would be a good time to think about getting rid of the concept of primary and secondary systems entirely. And instead just make it all be pools by which I mean users can simply make a pool, name it, select what storage devices are apart of it and what kind of disk management system they want that pool to use. Be it the unRAID Array, ZFS or BTRFS and the kind of RAID mode - Including XFS if they just want a single device pool without RAID. Then with the share system you can target any pool and select what mover does, being able to move from one pool to any other pool. And also select which pool should act as a cache for another pool. All generalised and standardised. I think this will be simpler for new users to grasp especially since a few updates ago we gained the ability to make multiple cache pools and now we're gaining ZFS and you're thinking about renaming unRAID array to Primary Pool and I assume that means other kinds of pools will be called Secondary Pools which if someone only wants ZFS and doesn't want to use unRAID array's will be a little confusing. I also think this is a great way to introduce the concept of multiple unRAID array's for users who want that. Since it's all pools you can just tell someone, sure make two pools and just set their modes to both be unRAID Arrays. Pretty simple in my mind.

You don't need to do a preclear on each drive however doing one can give you some confidence and usually if a drive is going to fail it does so at the very beginning of its usage or after a few years (the so-called bathtub curve as described by musclecups in the first post). So if you do a preclear what are the benefits? - You find out if a drive is bad straight away before you expand or rebuild your array. You find out it's bad during an easy return window where you can get an exchange directly from the seller as opposed to dealing with an RMA with the drives manufacturer (which could take more than a week vs a few hours to a day depending on where you bought the drive). Personally the way I do them is I do the pre-clear write and the post-clear read. I do not do the pre-read, I feel that's unnecessary personally. For the 18TB drives I buy it generally takes around 46 hours in total to do the write + post read. It does seem like a while doesn't it? but you can do multiple drives at once. In-fact I did 4 x 18TB drives at once without issue. Get a strong HBA and you can easily preclear 10+ drives at the same time without performance degradation. But also patience is key, especially with unRAID. I don't mind waiting for that extra knowledge that a drive is good. One added benefit of a pre-clear is you get to hear the drive acoustically running flat out for a few hours, sometimes drives will pass but may generate an annoying high pitched tone or a audible clicking noise, things that may give you pause about introducing it to your array.

Mhm, totally understandable. This chassis is really nice I'm sure you'll like it a lot, will make a great base for your server and all the staged upgrades

I do hope you will post your finished build on the forums, they're great to admire! I will definitely test the ZFS pool performance compared to BTRFS once it's officially available and if it's as-fast or faster I'll probably use it instead.

I have an update to the build today that may be of interest to others. I've installed 4 x 2TB Samsung 980 Pro's to use as a RAID0 cache (this is only for non-essential data). I came across an issue with my original 2TB cache in that it was too small. On some days I would put over 2TB into it requiring multiple moves in a single day and I also wanted the cache drive to hold its data for a few days to a week so that my software can interact with the data stored there in a faster manner than is possible with the unraid array itself. As a result I got the 4 x 2TB Samsungs and another PCIe card to place them and installed that into the server as shown below. And below is a picture of it installed into the server with my other add-in cards. You may notice there is no GTX 1080 Ti installed (mentioned on page 1) and that's because I don't need it at present and so I decided to remove it from the server as to extend the cards working life. With this addition to the server I decided to transition my previous cache drive (2TB SN850) into a 2nd drive for my VM's and Dockers and so I put both of them in RAID1. This was quite painless to do. Now I'm sure you're all curious about the performance of 4 x 2TB 980 Pro in RAID0.. as was I.. but there is an issue with the current 5.19 Linux Kernel when combined with EPYC CPU's which introduces a dummy I/O wait and that significantly curbs performance. This is a regression that wasn't present in earlier kernels and is already fixed in 6.x kernels and because of that I'm restricted to 2.1GB/s and this has been confirmed by other users that they are facing the same hard limits with the same kernel combined with EPYC. And that brings me to the performance screenshot below: Once the kernel issues are resolved I'll post an updated screenshot of how this really performs. Until then I hope this was at-least interesting. I really like these simple PCIe to M.2 adapter cards (which require bifurcation on your motherboard). The one I purchased 7 months ago is still working flawlessly and if you are buying M.2 drives which already have heatsinks as I am you can see in the above screenshot that the temperatures can be really good. As of now the server has 14TB of NVMe storage (1x2TB 970 Evo Plus, 2x2TB SN850, 4x2TB 980 Pro), quite a lot more than I initially thought I would use, I also have 10 x M.2 slots including two on the motherboard (1 unused) and 4 on each PCIe card (2 slots unused).

Were any of these releases (6.11.1, .2, .3, .4 or .5) put out as RC's before they became stable? - I feel like 14 days is reasonable considering the normal Unraid release schedule prior to the rapid fire we've had recently, would probably have resulted in only a .2 release being a thing with .3 .4 and .5 not having been necessary.

Might I suggest adding a setting to UnRAID where users can be part of Release Candidates or Stable releases. And all new releases have to stay as Release Candidates for 14 days before becoming Stable so issues like what this release fixes can be found by users who want the bleeding edge and so forth.

Here is where I gave my thoughts on the USB method in full, back in July:

Dear Jonathan, I already use Veeam agents inside my VM's to do backups inside the VM's. What I would like from Unraid is backing up of all the settings surrounding the VM's in the host itself, not of the VDISKS but of the settings and so forth. I'd also like snapshots but that's a separate thing. I'd also like proper Docker backups. I find myself just using a VM instead currently due to the backup situation. I mean.. what it would bring is redundancy against failure and a higher chance that the system will boot from cold without issue. That's kind of the point of RAID1, redundancy. I can also run pfSense from memory (it has the option) and yet I still run RAID1 drives there too. I'm fully aware that unRAID runs from memory and that doesn't change my view on this subject. However I'm not suggesting that the USB method should go away, just that we get some additional ways to install and boot unRAID for those of us who want it. I do think it feels wrong, yes. I think we will just have to agree to disagree on this subject. It's not OCD that is my rationale, it's that I want my freshest drive that is youngest and will (hopefully) last the longest to be filled before my older drives that I'm in the process of swapping out, I'll likely remove the oldest (and highest) 10TB drive soon to remedy this. I'm adding about 8-12TB a month currently but that will be increasing. I expect to have all my 10TB drives removed by the end of the year and replaced with 18TB models and then the system will also be at near capacity around the same time frame.

Bit of an update on the build as I've had it for almost two months now. HARD DRIVE FAILURE & REPLACEMENT One of the 10TB drives received 20,000 errors during the first parity read check. This was the last drive inserted into the system and hadn't yet received any data. This wasn't too unexpected due to it being 3.9 years old. Anyway I promptly removed it and replaced it with another 18TB drive from a WD MyBook, the drive inside was identical to the WD Elements I already shucked which was good. So the total setup now is 5 x 18TB and 6 x 10TB with dual-parity enabled. Right now it's filling nicely, I've gone from 43TB used to 48.1TB used already and that is continuing to grow, with some future plans I think it'll be near to 90TB used by January. I have decided to switch out the 10TB drives with 18TB ones before adding more drives to the array due to them getting old and one has now partially failed. I did decide to keep an 18TB drive pre-cleared on a shelf. One I've tested rigorously and know can be used as a replacement when I need it. So far the server is running wonderfully, extremely quiet still, can't tell it's on without the blinking lights, great cooling with all the drives remaining around 31-33c and the CPU not going much over 50c. Power consumption is also good, about 175 Watts 99% of the time with brief increases to 210-220 Watts during an array parity check. GENERAL STABILITY AFTER A MONTH I've not had any lockups or crashes so this hardware seems well at home running unRAID. SMB SHARE PERFORMANCE WOES WITH MACOS The only active problem I'm having with unRAID since the very beginning is its very poor macOS compatibility. The performance and stability of SMB shares from unRAID under the latest macOS (Monterey) are abysmal. Sometimes I can have days where it works perfectly and other days I'm seeing share disconnections, files taking 10-15 seconds to open, and videos being streamed to a media player pausing every few seconds. It's bad. These issues do not apply to SMB shares on Windows however where it works perfectly. I'm able to hit 5GB/s (40Gb/s) via SMB share to a Windows VM running on unRAID, I can only dream of speeds close to that on my Mac which tops out around 3.75MB/s (30Mb/s). From a lot of googling and discussing this with people on the unRAID discord, it appears to just be how it is and everyone experiences the same issues to differing degrees. This issue is quite annoying and I do hope this situation will be improved in the future as I wasn't aware of it before I started my unRAID journey and it isn't something I experienced with my previous setup which ran Windows Server 2012 R2. SMB shares from that server worked perfectly with macOS. VM NETWORK ISSUES & RESOLUTION One strange issue that I encountered with my VMs was their networking was acting strangely under high load. For instance, if I'm transferring files at near 10Gb/s I would have my VNC or RDP sessions (hosted from inside the VMs themselves) stutter, pause and disconnect. Similarly, any network services inside the VMs would experience the same problems under high network activity. I spent a long time investigating the problem, trying to narrow things down. The issue did go away when passing through a physical 10Gb network card to the VM but I wanted to use the network bridge created by unRAID for all of my VMs since I can't put in so many network cards for each VM that I wanted to operate. Well, it turns out the default "Network Model" selected by unRAID is called virtio-net and the way this works presents a lot of overhead. A virtualized NIC is used to handle packets in and out of the VM inefficiently when compared to the virtio (without the -net) method which interacts with the VM in host memory making it much more efficient. So essentially by switching from virtio-net to virtio as my network model I was able to eliminate the network degradation I was experiencing. No more did my VNC or RDP sessions collapse under heavy network use and in fact via iPerf I was able to raise network performance from 9.8Gb/s to 38Gb/s when testing between a Windows VM and the unRAID Host. I should mention there is a big warning on using virtio instead of virtio-net, and that is it may make the system unstable when using the bridge network it creates with both virtual machines and docker containers at the same time. I've yet to experience this but I'm not using Docker much, instead preferring to run things from virtual machines. MY GENERAL OPINION OF UNRAID AND WOULD I DO THIS AGAIN So far I do love unRAID. I have the sense it's very polished and the community has filled in the gaps where needed, plugins like the unassigned devices stuff, pre-clear plugins, and unbalance. All of these shore up the operating system where it's needed and I'm very happy with it. I for sure would build another in the future and I of course intend to keep this one. I don't have any regrets about choosing unRAID over TrueNAS. THINGS I WOULD LIKE TO SEE IMPROVED IN UNRAID I feel it needs a proper bulletproof backup system. Right now backing up Virtual Machines isn't simple, there is a plugin but it's broken and the maintainer is seemingly MIA. I feel this is important enough to include in the base system. The same situation with backing up Docker containers, there is a plugin but it requires your dockers to be powered down to perform the backups. If I'm hosting Plex for instance as a docker and I schedule backups to happen every day then my family is going to get quite annoyed if they're watching something when the backup occurs. Yes, this can be mitigated by performing backups at say 4 am but it can still happen and backups shouldn't be disruptive. I'd also like to see the SMB performance on macOS meet parity with Windows. I don't have these same issues when hosting SMB shares from Windows Server or TrueNAS so I'm confident in saying this is something the unRAID team need to resolve. The second to last thing I'd like to see improved is right now when you have dual parity you cannot move drives around without losing parity on your second drive. I understand this is a limitation based on how the number of drives is used as part of the parity calculation and I don't have a problem with that, but when you use the drive fill mode on a share (fills one drive at a time in sequence) you may have a drive lower down that you want to be filled before higher drives. For instance, I replaced a 10TB drive with an 18TB. But my shiny new 18TB drive is at the bottom of my disks now and so the fill method doesn't yet use this drive and won't do so for almost another 40TB of data. I'd prefer if the OS had a way to assign drives so we can swap them around virtually with a remapping table to their physical allocations. This way I could position all my 18TB drives together in the UI so the share fill method worked as I would prefer it. The last thing I'd like to improve is the boot method. I would appreciate the ability to boot the OS from RAID1 drives. I would 100% install two SATA SSDs in RAID1 just for a more reliable boot method. The USB key thing is convenient but unreliable and so I'd like an additional boot method (keep USB keys, but add normal SATA/NVMe booting with RAID1 capability, please!) So that's about it, not too much I'd change, the OS gets hundreds of things right to a handful of things wrong. This is a great batting average.

Indeed, you would need to import it either from China from the manufacturer directly or from the UK. Keep in mind only the UK distributor (XCase) includes the ATX PSU bracket, they appear to machine them locally and include them in the package themselves. Otherwise the case comes with two brackets for server power supplies only. Another option may be buying Supermicro cases, either used or new. They're quite popular and decent.

You can indeed go with that motherboard and it will work just fine with this chassis and backplane. The reason I ended up with the 9500-8i is really down to logistics and there's no reason you would need to buy such a card, the Supermicro board you linked to would work fine. To explain my circumstances, at the time I was planning out this build I was going to use the Asrockrack ROMED8-T2 which doesn't include a SAS capable chip on board. It just has two SFF-8643 connectors that can only be used with SATA drives via included breakout cables. So because I was planning to buy that motherboard I had to get a HBA to combine with it. My choices were to buy a used LSI/Broadcom 9200, 9300 or 9400 series card from ebay sellers or buy a 9500 brand new from a reputable local retailer with a 3 year warranty. There was a local store that had a Supermicro branded 9300-8i equivalent using the same 3008 chip as that motherboard you linked to but it was like £250 and I thought for another £150 I can get the 9500-8i instead which was the latest HBA model. So I essentially convinced myself into getting the 9500-8i instead of a 9300-8i HBA. Now at the time because I was so locked into the ROMED8-T2 (I had it on pre-order) I never even considered the Supermicro range of boards until after I found out the ROMED8-T2 I had ordered wasn't going to arrive (and in-fact it's still not available even now and retailers are saying September 30th as of today). So by the time I realised that I needed to order a different motherboard I already had the Broadcom 9500-8i in my possession and I thought well I like the card, I might as well order the Supermicro board (they have 4 SKU's of the same board with varying feature levels) that doesn't include the SAS disk controller from Broadcom since I already have the 9500-8i HBA now and it's two generations ahead of what Supermicro was including on their motherboard. It also meant I could choose the Supermicro motherboard which contains the 2 x PCIe x8 connectors which I could use for more SSD's later on. So that's why I ended up with this card. It's totally not necessary. I think in hindsight I still would go with the 9500-8i just because its newer, faster, lower heat and it only costs £150 more than going with the Supermicro board that has the 3008 chip on-board over the one that doesn't. But that's my thinking now in hindsight for my own situation, I'd still probably recommend others to save the £150 and get the Supermicro haha - More of a do as I say not as I do kinda situation I guess.

With reconstruct write enabled I get 250MB/s, with it disabled 100MB/s. That's direct to the array, no cache disk in use.

This will get filled before the end of the year and more 18TB drives will be added at that time. I may even replace some of the 10TB drives already present with 18TB drives, these 10TB drives have been running 24.7 for 3 years in my previous server which did not have the ability to spin down drives at all due to the striping it did so they saw a lot of miles.

Today I finally got my Unraid storage configured. It took a lot of time shuffling data around to move from my old server but today all finished. Which means I now have things setup as such: 2 x 18TB = Parity 2 x 18TB = Data 7 x 10TB = Data 2 x 2TB WD SN850 NVMe = Cache & VM's 1 x 2TB Samsung 970 EVO Plus = Probably a scratch drive for VM's not sure yet. I'm quite happy with Unraid so far. The array performance has actually been better than I expected. Admittedly I had low expectations and had ran it as a test on some other hardware to really test how it worked and the performance but even so it surpassed what I thought it would do. The 9500-8i HBA I bought has been a really good purchase able to run every disk maxed out without any performance issues which is as expected, I think I've only hit 1/5th of its total available bandwidth so far. That 2TB NVMe drive I've put in the server (Samsung Evo Plus) was originally going to be my disk cache drive but with the SN850's being so cheap I bought two of those and used one as my cache drive. I'm now not really sure what to do with the 2TB Evo Plus but perhaps it'll come in handy as a VM scratch drive or something else in the future.

Mmhm very true. When I hooked up my server to the UPS initially I was having communication errors, it would disconnect every few minutes and Unraid would bring up a notification about it losing communication. I thought okay maybe it doesn't support my UPS fully, but before I checked that I looked at the cables and it seems there's very little to no shielding on the included APC cable. Doesn't even have a ferrite choke. I noticed it was close to the AC power input on my server but not actually touching it, moved it further away and the communication problems were resolved. Just thought I'd add it to the topic incase someone else googles and comes across this problem

I've actually not tried it, I always just used NUT in the past, out of habit as I used to have a CyberPower UPS and it just worked, but I'll check out the APC stuff maybe it's better.

Indeed, there are a lot of people who don't understand all that, I am not one of those people.

My Virtual Machines shut down in less than 30 seconds (really it's like 10 seconds). And I have things configured to shutdown when the server has no power for more than 1 minute. So there will be ample time for everything to shutdown gracefully. I would expect the UPS to remain above 85%. This UPS is perfectly adequate for my usage. I understand you're having to make assumptions as people probably do all kinds of silly things and you've seen a lot of it but I already know not to run the battery down, I know it ages it faster, I'm not yet using a Lithium-ion equipped UPS that can do 500+ cycles. I never intended to run this UPS for 25 minutes in a power outage it was just a statement about the capacity. I've often run servers with 17 Minutes or less runtime projections without issue, it's important to know your hardware and how long it takes to shutdown. All that of course I know. I'll be moving to Network UPS Tools (NUT) soon I have other devices that will act as clients to shutdown that will be connected to the UPS as-well like my pfSense router and some other things I write myself can also act as clients. So don't worry, all covered, perfectly fine.

Today I hooked up the server to my UPS (the same APC model shown in the above post). And so now I have some power consumption numbers. The server idle consumes 150 Watts, under a parity rebuild with all Hard Disks involved it consumes 200 Watts. With a high CPU load (75% or so, so not completely maxed) it consumes 260 Watts. This is quite a solid showing for the energy efficiency. I was surprised as my old server which is a Dual-XEON E5-2667v2 with 16 DIMM's consumes about 360 Watts idle and can hit 500 Watts under a sustained CPU load. So this is less than half that server which is excellent. Connected to my UPS I have a whole bunch of devices including two switches (10Gb and 1Gb, both 16 ports), a WiFi access point, a Cloud Key, a pfSense router with a Quad Core i5 a CCTV camera. All of those devices combined consume 96 Watts and I made sure to monitor that over a full day before I connected the EPYC server so I could gauge exactly how much the server uses on its own. As a result of this idle the total power consumption is about 250 Watts (Server 150 + 100 for the other devices) which gives me a 25 minute runtime on the UPS (based on its own calculations) which is pretty good. Below I've included some screenshots of the UPS power draw as displayed in Unraid, remember this is Server + all those other devices. The load during this screenshot was a second parity drive being built while two VM's were running some medium loads.

I used the search and didn't see anyone mention the Broadcom 9500-8i HBA's. I recently purchased a 9500-8i for use with Unraid and it works really well under 6.10.3. Completely plug and play due to the Kernel having the appropriate driver and the performance is as you would expect, extremely fast. Just thought I'd post this as even on Google I didn't see more than one topic where someone said they had this card combined with Unraid so thought I'd chime in with a recommend.

Certainly you can pick my brain, anyone reading who wants some advice feel free to ask here, on discord or PM, I'm happy to help.