Baymax Build – My Ultimate Plex Server

ramblinreck47 · May 18, 2020

Objective:

I really wanted to replace my HP Z220 (i7-3770 and 16GB DDR3 Non-ECC). The system ran fine and was dependable, but I had maxed out all the hard drive bays, couldn’t get a HBA and P2000 to work at the same time, and wanted to move to a rack mounted solution. I just moved to a new city with a significantly better upload speed (from 10Mbps to 50Mbps), and within a few years, would be buying a house in an area with either Google or AT&T fiber (up to 1Gbps upload). In order to handle all this bandwidth, I was shooting for a system that could handle around 15+ 1080p 8Mbps transcodes while not being too power hungry. ECC wouldn’t be necessary since practically all the data was either movies or TV shows, but if I was going to spend all this money, it couldn’t hurt to go ahead with ECC. I also had just gotten a better paying job and paid off all my student loans, so I could afford to go a little overboard. I wanted this build to last me the next 5 years with little alternations being needed.

What I came up with:

Case – Supermicro CSE-836BE16-R920B (Modified) (~$500, eBay & Supermicro store)

--> I wanted the ability to use full height PCIE cards but didn’t want to commit to having 24 x 3.5” HDD bays largely being unused, so this fit perfectly in with that. I probably spent too much on getting this case supped up, but it’s built like a tank, functions exceptionally well, is relatively quiet, and meets all my needs. The case itself cost a little less than $400 with the 2 x PWS-920P-SQ’s and rear 2 x 2.5” hotswap module included. I then spent about $100 more upgrading all the fans; 3 x FAN-0074L4’s on the fan wall and 2 x FAN-0104L4’s on the back. The 0104L4’s were easy to take out of their green fan holders and screw into the preexisting holders on the back of my case. The 0074L4’s took a little more work to cut the webbing on the side of the fans so that they would fit properly. Just a couple of quick snips with some wire cutters. With the super quiet PSU’s and new fans installed, the whole case became very quiet.

CPU – Intel Xeon E-2278G (8 cores/16 threads) (~$500, Provantage)

--> So, why after already buying a Nvidia P2000 (at the wonderful price of $270!) did I go with an Intel/Quicksync build?! Well, it came down to Intel’s stability with Unraid and the iGPU. Although Ryzen CPU’s are fantastic processors at a very affordable cost, they’re still not 100% compatible/stable with Unraid. Now, I know they work well for most cases but there are still little quirks that either pop up due to AGESA versions, Unraid versions, and RAM peculiarities that make them not fully fleshed out. Intel, on the other hand, has a very stable and mature chipset that works great with Unraid in almost all cases. I wanted to be able to plug-n-play with a setup and not have to worry about crashes and sensors not working or being supported. The iGPU can be easily passed through to a docker container without needing to run an unofficial version of Unraid, like what would be needed to use the P2000. The E-2278G has the processing power to stay relevant for years to come, the official support for ECC (it’s not exactly known how ECC and Ryzen work together), and a top of the line iGPU that can handle a crazy amount of transcodes without drawing that much power. I wanted the E-2288G but couldn’t find one when I finally had the money to purchase it. I figured the E-2278G, for me, would be just as good and carry some of the same long-term value that the E-2288G would. I also thought about going with a E5-2680 v3 or E5-2690 v3 and P2000, but I just didn’t feel the need for more cores, more PCIE lanes, and lower single threaded performance. If the E-2278G becomes underpowered in the future, I’ll just sell the whole setup and start over with something new. I’ll probably switch to Ryzen somewhere down the line when that happens as the hardware and software for it becomes more mature and refined.

CPU Cooler – Noctua NH-D9L (~$60, Amazon)

--> I could have gone with a smaller cooler since the E-2278G only has a TDP of 80W but I wanted to go a little beefier just in case I upgraded to a more powerful processor on a different build sometime in the future. It’s also fits perfectly in a 3U case like the one I have, and its fan is very quiet. The installation was a breeze. Noctua really knows how to make this easy.

Motherboard – Supermicro X11SCH-F (~$300, Amazon)

--> Since I had a Supermicro chassis already and wanted to be able to use IPMI, this looked like a fantastic option. The X11SCA-F would have also been fine, but I wanted to have a separate LAN port for IPMI and didn’t need extra PCIE lanes. I could have also gone with the ASRock Rack E3C246D4U, but I’ve seen several bugs with that brand and motherboard and wasn’t sure it would be the best option. The Supermicro IPMI has been very nice and shows everything very well. The fans are controlled with the IPMI plugin in Unraid, and I’ve had no issues with them after getting the right settings.

Memory – 2 x 16GB Supermicro MEM-DR416L-HL01-EU26 (Hynix HMA82GU7CJR8N-VK 2666mhz ECC UDIMM) (~$180, Newegg)

--> This RAM was on the QVL for the motherboard and was at a decent price. I MEMTEST’d it when initially setting up the system and had no issues with it. I didn’t really need 32GB, but this will give me plenty of memory bandwidth for transcoding as well as any VM’s I decide to run in the future.

HBA – HP H220 (IT Mode) (~$55, eBay)

--> I had a LSI 9211-8i that I was already using in my HP Z220, but I figured I’d get a newer HBA that could use a PCIE 3.0 x8 slot to give more bandwidth to all 16 HDD’s when I eventually filled the case up. I bought a “new” one on eBay from seller jiawen2018. He filtered out as being located in the US but the payment definitely went to China. I was very hesitant to use this card, but I tested it out pretty thoroughly before switching the 9211-8i. After updating the firmware on the card, everything worked well, and I’ve been using it without issue for the last couple of weeks (even did a couple of parity checks). It’s one of those items that was really unnecessary since I only have 5400rpm drives.

Notes from using it for several months:

- Although the IPMI is nice, I’m finding that I don’t really need it all that much. I should be traveling right now for work but due to the virus, I’m largely stuck in the lab or at home. So, the possibility of needing it might change, but the server rack is very easy to get to in my house and it’s right beside my desk that has a mouse, keyboard, and monitor. I could have easily bought a simple KVM switch and been just as well off. In the future, I might forgo buying a motherboard with IPMI. It’ll save some money that I could use elsewhere.

- The iKVM also freezes whenever the i915 kernel loads in Unraid. It’s not all that serious since if I’m going to have any issues upgrading or rebooting my server, the problem will likely emerge before the boot process even gets that far. There must be a setting I’m missing somewhere to get it to stop freezing. Everything else in the BMC is operational though and the IPMI plugin is really nice in Unraid for making any sort of adjustments and monitoring all IPMI related activities.

- I don’t regret getting the E-2278G but I could have easily been happy with an E-2146G/E-2246G. It would have saved about $150, and I’d be getting roughly the same performance for what I do. The increased number of cores and threads will be useful if I ever start using VM’s but for right now, it’s way overkill.

- The E-2278G hasn’t been tested all that much so far, and the most transcodes I’ve had going at one time so far is 7 x 1080p 10GB-20GB to 720p 4Mbps. It handled all of them without issue, and I don’t think I’ve even seen the overall CPU usage go about 10% for practically any reason (maybe some single threaded docker containers might use more power but since this CPU is so powerful, it makes quick work of it). This thing is a monster.

- When starting up the server and everything going full force, it uses about 200W, but when the CPU and all 13 drives are sitting at idle, it averages about 140W.

- For some reason, the X11SCH-F won’t actually completely power down the system when I hit the Power Down button in Unraid. Unraid goes offline and for all accounts has a clean shutdown but everything except the motherboard stays powered on. I must physically go and hold the power button down on the chassis to get it to finish powering off. Even the BMC power down button won’t do anything. It’s not a big issue since I’ll be doing all future upgrades at home and if I have to restart the system after a power loss (once the UPS kicks back on), the Power On button in the BMC does work as intended. It’s just weird.

- Depending on how Unraid 6.9 stable and future releases work with Ryzen and if the P2000 goes on sale (it’s been below $300 before), I might end up selling this rig and making the transition over to Team Red. Since I only pay about $0.07/kWh, the power consumption difference between the two wouldn’t be that much and I’d have a lot more options for upgrading in the future. It turns out now that I’ve finished this build and it’s entirely stable, I’m somewhat bored and I like to tinker with things. We’ll see how things play out because this system is still robust and problem free. Also, I probably would drop IPMI and get a regular X570 motherboard. I’m not sure I’d go with ECC memory since the performance and reporting of it on Ryzen is suspect (link below). It depends on the availability of 3200mhz ECC UDIMM’s (they’re theoretically out there right now but impossible to get). If they don’t become purchasable at a decent price, I’ll just roll with some regular 3200mhz non-ECC UDIMM’s, memtest them, and save some money.

- Possible future upgrades for this build: bump up to another 32GB of RAM to give more bandwidth for transcoding and VM’s, replace my 2 x 500GB SSD’s with 2 x 500GB-1TB NVME 3.0 x4 drives, add in a 10Gbe card?... all these are completely unnecessary but that’s really what this build has always been about

Let me know if you have any questions about this setup. I’m more than willing to share everything I know.

Links:

Discussion about X470D4U and ECC reporting on Ryzen boards: https://www.ixsystems.com/community/threads/freenas-build-with-10gbe-and-ryzen.77752/

How I setup my fans in the IPMI plugin: reserved