Everything posted by Pri
-
Soon™️ 6.12 Series
haha never mind then. Funny thing is, I even reacted to that post when he made it originally.
-
Soon™️ 6.12 Series
Ya know, since you're thinking about changing the "unRAID array" to be a "Primary Pool" perhaps now would be a good time to think about getting rid of the concept of primary and secondary systems entirely. And instead just make it all be pools by which I mean users can simply make a pool, name it, select what storage devices are apart of it and what kind of disk management system they want that pool to use. Be it the unRAID Array, ZFS or BTRFS and the kind of RAID mode - Including XFS if they just want a single device pool without RAID. Then with the share system you can target any pool and select what mover does, being able to move from one pool to any other pool. And also select which pool should act as a cache for another pool. All generalised and standardised. I think this will be simpler for new users to grasp especially since a few updates ago we gained the ability to make multiple cache pools and now we're gaining ZFS and you're thinking about renaming unRAID array to Primary Pool and I assume that means other kinds of pools will be called Secondary Pools which if someone only wants ZFS and doesn't want to use unRAID array's will be a little confusing. I also think this is a great way to introduce the concept of multiple unRAID array's for users who want that. Since it's all pools you can just tell someone, sure make two pools and just set their modes to both be unRAID Arrays. Pretty simple in my mind.
-
Preclear
You don't need to do a preclear on each drive however doing one can give you some confidence and usually if a drive is going to fail it does so at the very beginning of its usage or after a few years (the so-called bathtub curve as described by musclecups in the first post). So if you do a preclear what are the benefits? - You find out if a drive is bad straight away before you expand or rebuild your array. You find out it's bad during an easy return window where you can get an exchange directly from the seller as opposed to dealing with an RMA with the drives manufacturer (which could take more than a week vs a few hours to a day depending on where you bought the drive). Personally the way I do them is I do the pre-clear write and the post-clear read. I do not do the pre-read, I feel that's unnecessary personally. For the 18TB drives I buy it generally takes around 46 hours in total to do the write + post read. It does seem like a while doesn't it? but you can do multiple drives at once. In-fact I did 4 x 18TB drives at once without issue. Get a strong HBA and you can easily preclear 10+ drives at the same time without performance degradation. But also patience is key, especially with unRAID. I don't mind waiting for that extra knowledge that a drive is good. One added benefit of a pre-clear is you get to hear the drive acoustically running flat out for a few hours, sometimes drives will pass but may generate an annoying high pitched tone or a audible clicking noise, things that may give you pause about introducing it to your array.
-
HYEPYC | High End EPYC Milan-X Build
Mhm, totally understandable. This chassis is really nice I'm sure you'll like it a lot, will make a great base for your server and all the staged upgrades
-
HYEPYC | High End EPYC Milan-X Build
I do hope you will post your finished build on the forums, they're great to admire! I will definitely test the ZFS pool performance compared to BTRFS once it's officially available and if it's as-fast or faster I'll probably use it instead.
-
HYEPYC | High End EPYC Milan-X Build
I have an update to the build today that may be of interest to others. I've installed 4 x 2TB Samsung 980 Pro's to use as a RAID0 cache (this is only for non-essential data). I came across an issue with my original 2TB cache in that it was too small. On some days I would put over 2TB into it requiring multiple moves in a single day and I also wanted the cache drive to hold its data for a few days to a week so that my software can interact with the data stored there in a faster manner than is possible with the unraid array itself. As a result I got the 4 x 2TB Samsungs and another PCIe card to place them and installed that into the server as shown below. And below is a picture of it installed into the server with my other add-in cards. You may notice there is no GTX 1080 Ti installed (mentioned on page 1) and that's because I don't need it at present and so I decided to remove it from the server as to extend the cards working life. With this addition to the server I decided to transition my previous cache drive (2TB SN850) into a 2nd drive for my VM's and Dockers and so I put both of them in RAID1. This was quite painless to do. Now I'm sure you're all curious about the performance of 4 x 2TB 980 Pro in RAID0.. as was I.. but there is an issue with the current 5.19 Linux Kernel when combined with EPYC CPU's which introduces a dummy I/O wait and that significantly curbs performance. This is a regression that wasn't present in earlier kernels and is already fixed in 6.x kernels and because of that I'm restricted to 2.1GB/s and this has been confirmed by other users that they are facing the same hard limits with the same kernel combined with EPYC. And that brings me to the performance screenshot below: Once the kernel issues are resolved I'll post an updated screenshot of how this really performs. Until then I hope this was at-least interesting. I really like these simple PCIe to M.2 adapter cards (which require bifurcation on your motherboard). The one I purchased 7 months ago is still working flawlessly and if you are buying M.2 drives which already have heatsinks as I am you can see in the above screenshot that the temperatures can be really good. As of now the server has 14TB of NVMe storage (1x2TB 970 Evo Plus, 2x2TB SN850, 4x2TB 980 Pro), quite a lot more than I initially thought I would use, I also have 10 x M.2 slots including two on the motherboard (1 unused) and 4 on each PCIe card (2 slots unused).
-
Unraid OS version 6.11.5 available
Were any of these releases (6.11.1, .2, .3, .4 or .5) put out as RC's before they became stable? - I feel like 14 days is reasonable considering the normal Unraid release schedule prior to the rapid fire we've had recently, would probably have resulted in only a .2 release being a thing with .3 .4 and .5 not having been necessary.
-
Unraid OS version 6.11.5 available
Might I suggest adding a setting to UnRAID where users can be part of Release Candidates or Stable releases. And all new releases have to stay as Release Candidates for 14 days before becoming Stable so issues like what this release fixes can be found by users who want the bleeding edge and so forth.
-
HYEPYC | High End EPYC Milan-X Build
Here is where I gave my thoughts on the USB method in full, back in July:
-
HYEPYC | High End EPYC Milan-X Build
Dear Jonathan, I already use Veeam agents inside my VM's to do backups inside the VM's. What I would like from Unraid is backing up of all the settings surrounding the VM's in the host itself, not of the VDISKS but of the settings and so forth. I'd also like snapshots but that's a separate thing. I'd also like proper Docker backups. I find myself just using a VM instead currently due to the backup situation. I mean.. what it would bring is redundancy against failure and a higher chance that the system will boot from cold without issue. That's kind of the point of RAID1, redundancy. I can also run pfSense from memory (it has the option) and yet I still run RAID1 drives there too. I'm fully aware that unRAID runs from memory and that doesn't change my view on this subject. However I'm not suggesting that the USB method should go away, just that we get some additional ways to install and boot unRAID for those of us who want it. I do think it feels wrong, yes. I think we will just have to agree to disagree on this subject. It's not OCD that is my rationale, it's that I want my freshest drive that is youngest and will (hopefully) last the longest to be filled before my older drives that I'm in the process of swapping out, I'll likely remove the oldest (and highest) 10TB drive soon to remedy this. I'm adding about 8-12TB a month currently but that will be increasing. I expect to have all my 10TB drives removed by the end of the year and replaced with 18TB models and then the system will also be at near capacity around the same time frame.
-
HYEPYC | High End EPYC Milan-X Build
Bit of an update on the build as I've had it for almost two months now. HARD DRIVE FAILURE & REPLACEMENT One of the 10TB drives received 20,000 errors during the first parity read check. This was the last drive inserted into the system and hadn't yet received any data. This wasn't too unexpected due to it being 3.9 years old. Anyway I promptly removed it and replaced it with another 18TB drive from a WD MyBook, the drive inside was identical to the WD Elements I already shucked which was good. So the total setup now is 5 x 18TB and 6 x 10TB with dual-parity enabled. Right now it's filling nicely, I've gone from 43TB used to 48.1TB used already and that is continuing to grow, with some future plans I think it'll be near to 90TB used by January. I have decided to switch out the 10TB drives with 18TB ones before adding more drives to the array due to them getting old and one has now partially failed. I did decide to keep an 18TB drive pre-cleared on a shelf. One I've tested rigorously and know can be used as a replacement when I need it. So far the server is running wonderfully, extremely quiet still, can't tell it's on without the blinking lights, great cooling with all the drives remaining around 31-33c and the CPU not going much over 50c. Power consumption is also good, about 175 Watts 99% of the time with brief increases to 210-220 Watts during an array parity check. GENERAL STABILITY AFTER A MONTH I've not had any lockups or crashes so this hardware seems well at home running unRAID. SMB SHARE PERFORMANCE WOES WITH MACOS The only active problem I'm having with unRAID since the very beginning is its very poor macOS compatibility. The performance and stability of SMB shares from unRAID under the latest macOS (Monterey) are abysmal. Sometimes I can have days where it works perfectly and other days I'm seeing share disconnections, files taking 10-15 seconds to open, and videos being streamed to a media player pausing every few seconds. It's bad. These issues do not apply to SMB shares on Windows however where it works perfectly. I'm able to hit 5GB/s (40Gb/s) via SMB share to a Windows VM running on unRAID, I can only dream of speeds close to that on my Mac which tops out around 3.75MB/s (30Mb/s). From a lot of googling and discussing this with people on the unRAID discord, it appears to just be how it is and everyone experiences the same issues to differing degrees. This issue is quite annoying and I do hope this situation will be improved in the future as I wasn't aware of it before I started my unRAID journey and it isn't something I experienced with my previous setup which ran Windows Server 2012 R2. SMB shares from that server worked perfectly with macOS. VM NETWORK ISSUES & RESOLUTION One strange issue that I encountered with my VMs was their networking was acting strangely under high load. For instance, if I'm transferring files at near 10Gb/s I would have my VNC or RDP sessions (hosted from inside the VMs themselves) stutter, pause and disconnect. Similarly, any network services inside the VMs would experience the same problems under high network activity. I spent a long time investigating the problem, trying to narrow things down. The issue did go away when passing through a physical 10Gb network card to the VM but I wanted to use the network bridge created by unRAID for all of my VMs since I can't put in so many network cards for each VM that I wanted to operate. Well, it turns out the default "Network Model" selected by unRAID is called virtio-net and the way this works presents a lot of overhead. A virtualized NIC is used to handle packets in and out of the VM inefficiently when compared to the virtio (without the -net) method which interacts with the VM in host memory making it much more efficient. So essentially by switching from virtio-net to virtio as my network model I was able to eliminate the network degradation I was experiencing. No more did my VNC or RDP sessions collapse under heavy network use and in fact via iPerf I was able to raise network performance from 9.8Gb/s to 38Gb/s when testing between a Windows VM and the unRAID Host. I should mention there is a big warning on using virtio instead of virtio-net, and that is it may make the system unstable when using the bridge network it creates with both virtual machines and docker containers at the same time. I've yet to experience this but I'm not using Docker much, instead preferring to run things from virtual machines. MY GENERAL OPINION OF UNRAID AND WOULD I DO THIS AGAIN So far I do love unRAID. I have the sense it's very polished and the community has filled in the gaps where needed, plugins like the unassigned devices stuff, pre-clear plugins, and unbalance. All of these shore up the operating system where it's needed and I'm very happy with it. I for sure would build another in the future and I of course intend to keep this one. I don't have any regrets about choosing unRAID over TrueNAS. THINGS I WOULD LIKE TO SEE IMPROVED IN UNRAID I feel it needs a proper bulletproof backup system. Right now backing up Virtual Machines isn't simple, there is a plugin but it's broken and the maintainer is seemingly MIA. I feel this is important enough to include in the base system. The same situation with backing up Docker containers, there is a plugin but it requires your dockers to be powered down to perform the backups. If I'm hosting Plex for instance as a docker and I schedule backups to happen every day then my family is going to get quite annoyed if they're watching something when the backup occurs. Yes, this can be mitigated by performing backups at say 4 am but it can still happen and backups shouldn't be disruptive. I'd also like to see the SMB performance on macOS meet parity with Windows. I don't have these same issues when hosting SMB shares from Windows Server or TrueNAS so I'm confident in saying this is something the unRAID team need to resolve. The second to last thing I'd like to see improved is right now when you have dual parity you cannot move drives around without losing parity on your second drive. I understand this is a limitation based on how the number of drives is used as part of the parity calculation and I don't have a problem with that, but when you use the drive fill mode on a share (fills one drive at a time in sequence) you may have a drive lower down that you want to be filled before higher drives. For instance, I replaced a 10TB drive with an 18TB. But my shiny new 18TB drive is at the bottom of my disks now and so the fill method doesn't yet use this drive and won't do so for almost another 40TB of data. I'd prefer if the OS had a way to assign drives so we can swap them around virtually with a remapping table to their physical allocations. This way I could position all my 18TB drives together in the UI so the share fill method worked as I would prefer it. The last thing I'd like to improve is the boot method. I would appreciate the ability to boot the OS from RAID1 drives. I would 100% install two SATA SSDs in RAID1 just for a more reliable boot method. The USB key thing is convenient but unreliable and so I'd like an additional boot method (keep USB keys, but add normal SATA/NVMe booting with RAID1 capability, please!) So that's about it, not too much I'd change, the OS gets hundreds of things right to a handful of things wrong. This is a great batting average.
-
HYEPYC | High End EPYC Milan-X Build
Indeed, you would need to import it either from China from the manufacturer directly or from the UK. Keep in mind only the UK distributor (XCase) includes the ATX PSU bracket, they appear to machine them locally and include them in the package themselves. Otherwise the case comes with two brackets for server power supplies only. Another option may be buying Supermicro cases, either used or new. They're quite popular and decent.
-
HYEPYC | High End EPYC Milan-X Build
You can indeed go with that motherboard and it will work just fine with this chassis and backplane. The reason I ended up with the 9500-8i is really down to logistics and there's no reason you would need to buy such a card, the Supermicro board you linked to would work fine. To explain my circumstances, at the time I was planning out this build I was going to use the Asrockrack ROMED8-T2 which doesn't include a SAS capable chip on board. It just has two SFF-8643 connectors that can only be used with SATA drives via included breakout cables. So because I was planning to buy that motherboard I had to get a HBA to combine with it. My choices were to buy a used LSI/Broadcom 9200, 9300 or 9400 series card from ebay sellers or buy a 9500 brand new from a reputable local retailer with a 3 year warranty. There was a local store that had a Supermicro branded 9300-8i equivalent using the same 3008 chip as that motherboard you linked to but it was like £250 and I thought for another £150 I can get the 9500-8i instead which was the latest HBA model. So I essentially convinced myself into getting the 9500-8i instead of a 9300-8i HBA. Now at the time because I was so locked into the ROMED8-T2 (I had it on pre-order) I never even considered the Supermicro range of boards until after I found out the ROMED8-T2 I had ordered wasn't going to arrive (and in-fact it's still not available even now and retailers are saying September 30th as of today). So by the time I realised that I needed to order a different motherboard I already had the Broadcom 9500-8i in my possession and I thought well I like the card, I might as well order the Supermicro board (they have 4 SKU's of the same board with varying feature levels) that doesn't include the SAS disk controller from Broadcom since I already have the 9500-8i HBA now and it's two generations ahead of what Supermicro was including on their motherboard. It also meant I could choose the Supermicro motherboard which contains the 2 x PCIe x8 connectors which I could use for more SSD's later on. So that's why I ended up with this card. It's totally not necessary. I think in hindsight I still would go with the 9500-8i just because its newer, faster, lower heat and it only costs £150 more than going with the Supermicro board that has the 3008 chip on-board over the one that doesn't. But that's my thinking now in hindsight for my own situation, I'd still probably recommend others to save the £150 and get the Supermicro haha - More of a do as I say not as I do kinda situation I guess.
-
HYEPYC | High End EPYC Milan-X Build
With reconstruct write enabled I get 250MB/s, with it disabled 100MB/s. That's direct to the array, no cache disk in use.
-
HYEPYC | High End EPYC Milan-X Build
This will get filled before the end of the year and more 18TB drives will be added at that time. I may even replace some of the 10TB drives already present with 18TB drives, these 10TB drives have been running 24.7 for 3 years in my previous server which did not have the ability to spin down drives at all due to the striping it did so they saw a lot of miles.
-
HYEPYC | High End EPYC Milan-X Build
Today I finally got my Unraid storage configured. It took a lot of time shuffling data around to move from my old server but today all finished. Which means I now have things setup as such: 2 x 18TB = Parity 2 x 18TB = Data 7 x 10TB = Data 2 x 2TB WD SN850 NVMe = Cache & VM's 1 x 2TB Samsung 970 EVO Plus = Probably a scratch drive for VM's not sure yet. I'm quite happy with Unraid so far. The array performance has actually been better than I expected. Admittedly I had low expectations and had ran it as a test on some other hardware to really test how it worked and the performance but even so it surpassed what I thought it would do. The 9500-8i HBA I bought has been a really good purchase able to run every disk maxed out without any performance issues which is as expected, I think I've only hit 1/5th of its total available bandwidth so far. That 2TB NVMe drive I've put in the server (Samsung Evo Plus) was originally going to be my disk cache drive but with the SN850's being so cheap I bought two of those and used one as my cache drive. I'm now not really sure what to do with the 2TB Evo Plus but perhaps it'll come in handy as a VM scratch drive or something else in the future.
-
HYEPYC | High End EPYC Milan-X Build
Mmhm very true. When I hooked up my server to the UPS initially I was having communication errors, it would disconnect every few minutes and Unraid would bring up a notification about it losing communication. I thought okay maybe it doesn't support my UPS fully, but before I checked that I looked at the cables and it seems there's very little to no shielding on the included APC cable. Doesn't even have a ferrite choke. I noticed it was close to the AC power input on my server but not actually touching it, moved it further away and the communication problems were resolved. Just thought I'd add it to the topic incase someone else googles and comes across this problem
-
HYEPYC | High End EPYC Milan-X Build
I've actually not tried it, I always just used NUT in the past, out of habit as I used to have a CyberPower UPS and it just worked, but I'll check out the APC stuff maybe it's better.
-
HYEPYC | High End EPYC Milan-X Build
Indeed, there are a lot of people who don't understand all that, I am not one of those people.
-
HYEPYC | High End EPYC Milan-X Build
My Virtual Machines shut down in less than 30 seconds (really it's like 10 seconds). And I have things configured to shutdown when the server has no power for more than 1 minute. So there will be ample time for everything to shutdown gracefully. I would expect the UPS to remain above 85%. This UPS is perfectly adequate for my usage. I understand you're having to make assumptions as people probably do all kinds of silly things and you've seen a lot of it but I already know not to run the battery down, I know it ages it faster, I'm not yet using a Lithium-ion equipped UPS that can do 500+ cycles. I never intended to run this UPS for 25 minutes in a power outage it was just a statement about the capacity. I've often run servers with 17 Minutes or less runtime projections without issue, it's important to know your hardware and how long it takes to shutdown. All that of course I know. I'll be moving to Network UPS Tools (NUT) soon I have other devices that will act as clients to shutdown that will be connected to the UPS as-well like my pfSense router and some other things I write myself can also act as clients. So don't worry, all covered, perfectly fine.
-
HYEPYC | High End EPYC Milan-X Build
Today I hooked up the server to my UPS (the same APC model shown in the above post). And so now I have some power consumption numbers. The server idle consumes 150 Watts, under a parity rebuild with all Hard Disks involved it consumes 200 Watts. With a high CPU load (75% or so, so not completely maxed) it consumes 260 Watts. This is quite a solid showing for the energy efficiency. I was surprised as my old server which is a Dual-XEON E5-2667v2 with 16 DIMM's consumes about 360 Watts idle and can hit 500 Watts under a sustained CPU load. So this is less than half that server which is excellent. Connected to my UPS I have a whole bunch of devices including two switches (10Gb and 1Gb, both 16 ports), a WiFi access point, a Cloud Key, a pfSense router with a Quad Core i5 a CCTV camera. All of those devices combined consume 96 Watts and I made sure to monitor that over a full day before I connected the EPYC server so I could gauge exactly how much the server uses on its own. As a result of this idle the total power consumption is about 250 Watts (Server 150 + 100 for the other devices) which gives me a 25 minute runtime on the UPS (based on its own calculations) which is pretty good. Below I've included some screenshots of the UPS power draw as displayed in Unraid, remember this is Server + all those other devices. The load during this screenshot was a second parity drive being built while two VM's were running some medium loads.
-
Recommended controllers for Unraid
I used the search and didn't see anyone mention the Broadcom 9500-8i HBA's. I recently purchased a 9500-8i for use with Unraid and it works really well under 6.10.3. Completely plug and play due to the Kernel having the appropriate driver and the performance is as you would expect, extremely fast. Just thought I'd post this as even on Google I didn't see more than one topic where someone said they had this card combined with Unraid so thought I'd chime in with a recommend.
-
HYEPYC | High End EPYC Milan-X Build
Certainly you can pick my brain, anyone reading who wants some advice feel free to ask here, on discord or PM, I'm happy to help.
-
HYEPYC | High End EPYC Milan-X Build
It'll be used for storing media, running virtual machines, running docker containers. The VM's will consist of stuff for my work, a mirror cluster of our live infrastructure made up of multiple VM's. That's mainly why it has so much memory and CPU cores. I'll of course also be running things like Plex. It'll be our home server running Homebridge and all our home automation stuff, CCTV recording and so on. The 1080 Ti will be used for machine learning applications with software that I write. I also have an RTX 3090 which is in my desktop system but I may put that into the server when I upgrade my desktop GPU. I really like the 24GB of VRAM for machine learning. The HBA comes already flashed in IT mode, I did flash it again to upgrade the firmware as mine came with v14 and v23 was out. It was very easy to flash, just download the file, one command in Windows from the StorCLI software and it was up to date. I'm very happy with the card, it runs very cool and low energy (5.9 Watts peak vs 11 Watts for the previous generation card for instance). The only downside apart from the price of the card is the availability of compatible cables. I had to use a 1 meter long SAS cable because I literally couldn't acquire a 0.5-0.6m one which would have fit better. This is due to it using the newer SlimSAS x8 port instead of MiniSAS HD so the cable choices are a bit more limited.
-
HYEPYC | High End EPYC Milan-X Build
In early June I started acquiring hardware for this build and I only just completed it this week with many parts taking an extremely long time to arrive. Now that it's complete I'm excited to share it with you after having enjoyed looking at your systems here on the forum and on the Unraid discord. Full Specifications CPU: AMD EPYC Milan 7443p 24 Core / 48 Thread 2.85 GHz base, 4 GHz boost (Removed early 2025) CPU: AMD EPYC Milan-X 7773X 64 Core / 128 Thread 2.2GHz base, 3.5GHz boost (Added early 2025) Heatsink: SuperMicro 4U Active SP3 cooler (Removed early 2024) Heatsink: Arctic 4U-M SP3 cooler (Added early 2024) RAM: Samsung 256GB (8x32GB) DDR4 RDIMM ECC 3200MHz CL22 (Removed early 2025) RAM: Samsung 1TB (8x128GB) DDR4 LRDIMM ECC 3200MHz CL22 (Added early 2025) MOBO: SuperMicro H12SSL-NT EPYC 7002/7003 support with dual 10Gb PSU: Corsair HX1200 Platinum 1200 Watt (Platinum Rated) HBA: Broadcom 9500-8i HBA (PCIe 4.0 NVMe/SAS/SATA) M.2 Card: Generic PH44 4 x M.2 PCIe 4.0 card M.2 Card2: Asus Hyper M.2 PCIe 4.0 card (Added Mid 2023) SSD1: Samsung PCIe 3.0 2TB 970 Evo Plus NVMe Drive SSD2 and 3: Western Digital PCIe 4.0 2TB SN850 NVMe Drive (Heatsink Sku) SSD4, 5, 6 and 7: Samsung PCIe 4.0 2TB 980 Pro NVMe Drives (Heatsink Sku) (Added Early 2023) SSD8, 9, 10 and 11: Kioxia CD6 U.3 PCIe 4.0 7.68TB NVMe Drives (Added Mid 2024) NIC: Intel PCIe 2.1 X550-T2 2x10Gb/s Ethernet Network Card (Added Mid 2023) NIC: Intel PCIe 2.1 X710-DA4 4x10Gb/s SFP+ Ethernet Network Card (Added Mid 2023) GPU: NVIDIA RTX A4000 (Added Feb 2024) HDD1: Western Digital 4x18TB SATA Hard Drives (Shucks) HDD2: Seagate 7x10TB SATA Hard Drives (NAS Editions) CASE: Gooxi 24-bay Chassis with 12Gb SAS3 Expander chip by PMC-Sierra CABLES: Custom Molex Cables from CableMod (Backplane<->PSU) FANS: Noctua, 3 x 120mm (A12x25), 2 x 60mm (A6x25) UPS: APC BR1600SI (960 Watts / 1.6kVA, Pure Sinewave) Early 2023 Edit: A few months later some changes have happened with the build. 4 x 2TB 980 Pro SSD's were added. You can view photos and information about that in this update post. Early 2024 Edit: Some more changes, new network cards, new graphics card and new CPU heatsink. You can view more about those things including photos in this update post. Mid 2024 Edit: Yet more changes, 4 x 7.68TB Kioxia SSD's were added. You can view photos and information about that in this update post. Early 2025 Edit: The RAM was swapped for 8 x 128GB 3200MHz LR-DIMM's from Samsung. You can view photos and information in this update post. The CPU was also swapped for a 64-core 7773X. You can view photos and info in this update post. Originally some of this hardware was different, for instance the motherboard was going to be the Asrock ROMED8-T2, the 18TB hard drives were going to be Toshiba enterprise drives, there was only going to be one SN850 SSD and the UPS was going to be a CyberPower model. But as prices of things changed and availability of certain things became problematic the build had to change. I wouldn't normally buy external hard drives and shuck them just due to the time and effort but they were simply too cheap to ignore at £222.99 each on Amazon Prime Day. With this pricing I was able to purchase 4 x 18TB externals from WD for less than the price of 3 x 18TB raw drives from Toshiba, WD or Seagate. An extra drive and at a lower cost? hard to pass up and as I show later they work perfectly. I linked to each individual part above in-case you want to see more information about a specific component used in the build. Individual Component Photos Almost everything was photographed but I cannot link them as small thumbnails in the thread so I'll spare you having to scroll past those and we'll begin with the completed build photos instead and I'll include a few of the more interesting component shots after those. The completed build The below photo was just after I initially built it with the stock fans, I hadn't attempted to do any cable management yet. The below photo is after the Noctua Fans have been installed and I did do a little bit of cable management. This is is why the fan cables have switched from the tomato-ketchup kind to the black fully sleeved kind. I did have to give-up the hotswap system that the case came with when changing fans but the Noctuas I purchased come with a 1cm long PWM connector from the fan and then a 30cm extension cable so it serves the same purpose. I could have cut the cables on the original fans and soldered the 30cm Noctua extensions to the hotswap PCB to maintain this functionality but I didn't want to do that preferring to keep the original fans as-is in-case the cases backplane had a fault requiring the case to be sent back to the retailer. Still the Noctuas work great in the original holders and the sliding mechanism is maintained. Individual Component Photos The case came with three of these 40mm thick industrial hotswap fans shown below that each consume 20 watts at peak output. They are extremely loud and move an incredible amount of air. I quickly swapped these out for Noctua Black Chromax A12x25's which are essentially silent and move around 40% of the air of these fans that came with the case. I'm happy to say the temperatures are still great with the slower Noctua fans. I did get two of the SN850 2TB's with Heatsink but unfortunately the Motherboards M.2 slots are too close together and only one can be installed. Thankfully I do have a 4 x NVMe PCIe card with wider spacing and so I'll be placing these two and my Samsung NVMe drive into that once the parity on my server is completed so I can power it down to add the card in. UPDATE: And the SN850's are now in their PCIe card which I've put into the server today. Below is a photo of those installed into the card. As you can see with these heatsinks the spacing is quite tight and SuperMicro unfortunately didn't take that into account when they put their slots right next to each other with only a single sheet of paper gap between. The card I'm using here is just a cheap generic model for £36, it comes with no fan or heatsink and requires Bifurcation support on your motherboard. I chose to buy this as I knew all the SSD's I'd be installing would have heatsinks on them and my server case has quite high airflow. Samsung likes to ship their memory individually wrapped. I've had this memory for about two months so I didn't open them when I took the photo in June as to keep them in pristine condition but you can see a better photo of them installed in the motherboard above. (Edit Early 2025, this RAM has since been replaced with 8 x 128GB of LRDIMM's from Samsung for 1TB of total system memory. You can view photos of that update here) The 7443p CPU below was chosen not just for its 24 Cores / 48 Threads and Zen3 Architecture but for its high clock speeds. It has a 2.85GHz base and 4GHz boost clock which is perfect for my use case. (Edit Early 2025: This processor was swapped for a 7773X 64-core, you can read more about that in this update post) It sure has a lot of pins, 4092 to be exact. If you know these EPYC chips they feature 129 PCIe 4.0 lanes (128 usually accessible to the user through slots and connectors and 1 reserved for the motherboard vendor to use with a BMC implementation). And eight memory channels which can take 3200MHz RDIMM's which is what I installed to guarantee I get the maximum infinity fabric bandwidth. The Broadcom 9500-8i below was chosen for a few reasons, I wanted to be able to use all 24 slots on my chassis at near line speed, this card can do that at 96Gb/ps (12GB/s) where each slot would have 500MB/s capability. It is a PCIe 4.0 x8 card and features signed firmware. I could have obtained a 9400-8i or even a 9300-8i on ebay for 1/3rd to 1/5th the cost of this card but I felt I wanted to get a brand new card at retail with a warranty and secure code signing for its firmware. This is thus a retail Broadcom card purchased from a reputable etailer. Below is my Intel X540-T2 ethernet card. I actually have several of these and also a X710-T4 which is a newer model with four ports. The reason I'm installing this into the server even though the motherboard already features 2 x 10Gb ethernet is because I'm going to be running pfSense in a VM on this server as a backup in the event my physical pfSense system has a hardware failure (which it did last year and it was very annoying). Early 2024 Edit: This X540-T2 card was subsequently removed from the server in mid-2023 and replaced with an X550-T2 which looks essentially identical. I did this change because the X550-T2 supports 2.5GbE and 5GbE in addition to 1GbE and 10GbE. I needed the 2.5GbE for a cable modem, thus the swap. So I'll be setting up a pfSense VM on Unraid and passing this card through just so I can have a backup since we all need internet and I work from home making it extra important. Some may be surprised that I went with an ATX power supply instead of a dual-redundant setup. This Chassis actually comes with several brackets which support single (ATX), dual and triple (server) redundant power supplies. I went with the HX1200 because while it's more power than I'll need in this server it has a 10 year warranty, platinum rated efficiency and it's silent upto 450 watts and very quiet beyond that. In-fact the main reason I even chose to build this server myself as opposed to buying a prebuilt system was because I wanted an ATX power supply so that the system could be as quiet as possible. And finally I thought I'd show the APC UPS, not really that exciting but it is a newer available model at-least in the UK. Features line interactivity and a pure sinewave which is important to me for stability. So what has been the damage for all of this. Everything listed in the spec sheet was purchased specifically for this build except for the NVIDIA GTX 1080 Ti, Intel X540-T2 and 7 x 10TB Hard Drives which I already had and will be moving from my previous build. So not including those parts the total cost has been £7,100 GBP or $8,604.13 USD. I imagine with those added parts included it would go to around £8,500-£9,000 making it around $11,000 USD. I did set out to build a very high end server and that certainly came with a price especially considering I wasn't prepared to compromise much by going with slower or less memory, an older generation CPU, a used chassis, an older HBA etc. If you're looking to build something similar you can probably obtain 75-80% the same capability and performance for half the cost by purchasing one generation older hardware and used parts where it makes sense and I would certainly advise you to do so! Feel free to ask any questions and I'll leave you with a couple system screenshots and benchmarks. Above: Some benchmarks ran from Windows before Unraid was installed on the system. Below: So many threads! Below: Building Parity before I insert the rest of my 7 x 10TB drives (I still need to copy files from that old server to this one before moving those drives over).