I’m currently running Unraid on an old Supermicro X9DRi-LN4+ with dual Intel Xeon E5-2670 v2 CPUs. It’s served me well for years but this old Supermicro is loud, hot, and power-hungry. I’m ready for an upgrade, and I’m looking for something that sips power on idle but can unleash the beast when it needs to. The heaviest load on my server is usually Plex transcodes (up to 6 simultaneous), but I also run 20+ other docker containers including Immich, Nextcloud, Matrix, Home Assistant, Mealie, Vaultwarden, etc.

 

Here is my current tentative plan:

CPU: Intel Core i5-14500 2.6 GHz 14-Core Processor
CPU Cooler: Noctua NH-D15 chromax.black 82.52 CFM CPU Cooler

CPU Cooler: BeQuiet! Dark Rock Elite

Motherboard: ASUS Pro WS W680-ACE
Memory: 2 x Kingston KSM48E40BD8KM-32HM 32 GB (2 x 32 GB) DDR5-4800 CL40 ECC Memory

Memory: 2 x Kingston KSM48E40BD8KI-32HA 32 GB (2 x 32 GB) DDR5-4800 CL40 ECC Memory
Cache Storage: 2 x Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive

Cache Storage: 2 x WD Black SN850X 4 TB M.2-2280 PCIe 4.0 X4 NVME SSD

Cache Storage: 2 x Crucial T700 4TB Gen5 NVMe M.2 SSD
Case: Fractal Design Meshify 2 XL ATX Full Tower Case + 5 x HDD Tray Kits
Power Supply: Corsair HX1200 Platinum 1200 W 80+ Platinum Certified Fully Modular ATX Power Supply

Power Supply: FSP Hydro Ti Pro 1000W Titanium

Power Supply: Seasonic Vertex PX-750 80+ Platinum Power Supply
Case Fan: 5 x Noctua A14 PWM chromax.black.swap 82.52 CFM 140 mm Fan

HBA: LSI 9400-16i

HBA: LSI 9201-8i

Network Card: TRENDnet 10 Gigabit PCIe SFP+ Network Adapter

Network Card: Intel X710-DA2 Dual 10Gbps SFP+

 

Main array storage is 10 x 10TB Seagate Exos drives, and would be moved over from my current build. I picked the 14500 because it is a re-badged Alder Lake and doesn't suffer from the recent microcode issues, and that expensive W680-ACE motherboard for the ECC RAM support.

 

My current Suprmicro server idles at 200W, so I imagine this would be quite a reduction on my power bill without compromising anything. I'd love to hear thoughts & suggestions.

Edited November 4, 20241 yr by Gazeley
changed SSD to Crucial T700

I have a similar system.

 

I opted for a 14700 CPU despite the microcode problem. I assume that Intel has solved the problem with the code 0x12B. Otherwise, the CPUs now have a 5-year guarantee.

 

Your CPU cooler is actually oversized. Noctua even has a tool that gives a recommendation for every CPU. For your CPU, the Noctua NH-D12L cooler is sufficient, which is also available in black and is cheaper.

 

The RAM you chose is outdated and discontinued. It was replaced by KSM48E40BD8KI-32HA. I have this too and it works without problems.

 

6 fans for the case are also a bit much and cost energy in addition to the purchase. First, create an air flow from front to back and look at the temperatures first. Then you can always upgrade.

 

Your power supply is absolute overkill. This then causes further unnecessary losses in idle.

 

Note: With the W680 boards you won't get above C3, but that didn't matter to me. In my setup with 3 x NVMe, 1 x Google TPU, 4 x HDD and no expansion card, with Frigate as the base load, I get 50W in idle.

4 hours ago, enect said:

Your CPU cooler is actually oversized. Noctua even has a tool that gives a recommendation for every CPU. For your CPU, the Noctua NH-D12L cooler is sufficient

 

Thanks for the tip! I swapped that part onto my build list.

 

4 hours ago, enect said:

The RAM you chose is outdated and discontinued. It was replaced by KSM48E40BD8KI-32HA. I have this too and it works without problems.

 

Good to know! Apparently I need to stop relying so heavily on PCpartpicker, their database is missing a lot of this server-grade stuff. Changed to this RAM on my plan.

 

4 hours ago, enect said:

6 fans for the case are also a bit much and cost energy in addition to the purchase.

 

I'm planning to fill the case out with 16 HDDs + 2 NVME SSDs. So I feel like I'm definitely going to want the 3 front fans to maximize airflow over the disks, and obviously I'll need at least 1 exhaust. I guess I could hold off on the 2 top fans until I see the temps, but I figured the power draw from 2 more fans would be negligible, and I'd rather err on the side of having too much cooling rather than not enough.

 

4 hours ago, enect said:

Your power supply is absolute overkill. This then causes further unnecessary losses in idle.

 

Interesting, I'm loosely basing this off builds like this, and they all have at least 1000W PSUs (because of all the HDDs I assumed). I'm sure in Unraid it will be very rare for all the HDDs to be spun up at once, but I want to make sure that if that happens it can handle everything. I also wanted some overhead so that if I add a GPU or something in the future I don't need to swap the whole PSU.

 

What PSU size would you recommend?

 

4 hours ago, enect said:

With the W680 boards you won't get above C3

 

You lost me here. What is C3?

 

4 hours ago, enect said:

I get 50W in idle

 

Going from my current system's 200W idle down to 50W, while also having a more powerful machine, would be amazing. Thank you for your input!

22 minutes ago, Gazeley said:

power draw from 2 more fans would be negligible

correct, two fans only make about 3-5 watts

My system runs with 2 front fans, the CPU cooler and a back fan. But I only have 4 HDDs and the NVMe's have a small cooling block.

 

25 minutes ago, Gazeley said:

What PSU size would you recommend?

Of course, it really depends on what you ultimately want to do with the system. But 1000W isn't even really worth it if you put an RTX 4090 in it and want to go all-out gaming with it. There are power supply calculators e.g.: https://www.bequiet.com/en/psucalculator

Without a graphics card, the typical small power supplies with 550W should be sufficient (e.g. be quiet! Pure Power 11 FM 550W). That also covers a typical server GPU. With a gaming GPU, it just needs a bit more. You can't say in general terms, as everyone uses their server differently. But 1000W just sounds like it's better to have than need.

 

33 minutes ago, Gazeley said:

You lost me here. What is C3?

Intel PowerStates. If someone wants to save as much power as possible, it's interesting. There are probably hundreds of posts about it in the forum. If you want to go with Intel, ECC and IPMI, which was also important to me... it doesn't matter to you. All alternative boards are on the same level/power consumption.

Someone measured this in a German forum: https://www.hardwareluxx.de/community/threads/vergleich-leistungsaufnahme-ecc-fähiger-lga-1700-mainboards.1339804/

That PSU calculator is very helpful:

 

 

Looks like 800W would be sufficient (assuming I add a GPU someday). I doubt there would ever be a scenario where all my HDDs are spinning AND I'm maxing out the CPU & GPU at the same time, so realistically I should never hit that max load.

 

So I think 800W would be ok, especially if I don't add a GPU - although I may want to go with 1000W just to be safe, and give me headroom to add something like a RTX 3080 if I want to self-host an LLM or something someday. Either way you're right that 1200W seems to be overkill.

 

Thank you again for taking the time to help!

6 hours ago, enect said:

This then causes further unnecessary losses in idle.

 

I did some reading and I understand what you mean here now. I've only ever built gaming PCs where I didn't give a shit about idle power use, so this is all sort of new territory for me.

 

If I'm hoping to have the server idling around 50watts or so, then it would be down at like 78% efficiency most of the time on a 1000W PSU:

 

Whereas if I went with a 750W PSU it would be around 83% efficiency at 50W:

 

Although that only seems to go so far. The low-power efficiency of the RM550x looks equal to, if not worse, than the HX750i above at 50W:

 

Anyway, now I see the reasoning behind going with a lower powered PSU and just upgrading it as needed if I ever do decide to add a GPU. I don't want to be running at a lower efficiency for potentially years and years on the off chance I might add a GPU someday.

 

I ran another calculator and it looks like I could get away with a 550W PSU:

 

But since the HX750i looks more efficient at low loads I think I'll go with that.

 

These are exactly the considerations when choosing the right PSU for a server. The Corsair would be a great choice, but you have to get it first. They aren't actually manufactured anymore, as far as I know. There is even a plug-in for the power supply that you can use to display the power consumption in the dashboard. 
 

In my humble opinion, you are on the right track and this will certainly be a nice, powerful server with acceptable power consumption. Have fun building it.

If you can find them, I believe the 750w seasonic titanium rated PSU is among the better ones sort-of still available at low power levels, and the 850 only slightly behind. Also top tier supplies. You will spin up all the drives at the same time at boot.

 

For power efficiency, consider sticking with intel x710 family SFP+/10g. Well priced on the used market. May not be significant with what is noted about c states here though.

Also evaluate whether you need that HBA card or can get away with onboard SATA and a more boring sata adapter or two - it may also keep power consumption down. I use a 6-port asm1166 m.2 adapter and shaved over 10w vs 8 port 9200-8i, no observed loss in throughput. Only 5 spinners for me, but 5 SSD for file shares (1 is just music for my squeezeserver, 4 are raid10 for array write cache and active/fast shares. Shares are backed up to the array still (plus more layers of backup)

1 hour ago, _cjd_ said:

I believe the 750w seasonic titanium rated PSU is among the better ones sort-of still available at low power levels, and the 850 only slightly behind. Also top tier supplies.

Wow you aren't kidding, the Seasonic Prime 750 Titanium appears to have amazing efficiency! Looks like I can still get them new from eBay or refurbished from Newegg. Changed my PSU out to that on the build list.

 

1 hour ago, _cjd_ said:

For power efficiency, consider sticking with intel x710 family SFP+/10g. Well priced on the used market.

Smallest I can find in that family is this Intel X710-DA2 with dual ports. You think that would still be more power efficient than something like this (Intel 82599EN) with a single port?

 

1 hour ago, _cjd_ said:

Also evaluate whether you need that HBA card or can get away with onboard SATA and a more boring sata adapter or two - it may also keep power consumption down.

I'll look into the motherboard specs and consider that. Thanks for the tip!

Edited October 29, 20241 yr by Gazeley

8 hours ago, Gazeley said:

Smallest I can find in that family is this Intel X710-DA2 with dual ports. You think that would still be more power efficient than something like this (Intel 82599EN) with a single port?

It's definitely better than older Intel chipsets, and even in my AMD system (x570/5600x) slightly better than mellanox connectx-3. I'd guess yes, but don't have data to back that up. At worst it won't be worse.

I posted a thread over on level1techs and got some great feedback. Made the following changes per their suggestions:

• i5-14500 changed to Ryzen 9 7900

(They convinced me that a Ryzen 7900 + GPU is better bang for my buck and has more ECC mobo options.)

• Noctua DH-D12L changed to BeQuiet! Dark Rock Elite

(Better cooling and quieter.)

• ASUS Pro WS W680-ACE changed to a ASUS ProArt X670E-CREATOR

(AM5 Socket, ECC Support)

• Samsung SSDs changed to WD Black SSDs

(Apparently Samsung has had some firmware issues with recent SSDs.)

• Seasonic TX 750 changed to FSP Hydro Ti Pro 1000W Titanium

(Better efficiency in general, but particularly in the 20-80W range.)

• TRENDnet 10G SFP+ changed to Intel X710-DA2 Dual 10Gbps SFP+

(Better/newer NIC)

 

This means I'll have to add a GPU for Plex transcoding. Looks like the RTX 3060 (w/ 12GB RAM) can handle ten 4K to 1080p transcodes, so that's currently my top contender.

Edited October 30, 20241 yr by Gazeley

Losing the Intel for transcoding may not be worth the trade, especially looking at power efficiency. Then again, it may be you'd need separate graphics for 6 transcode streams anyway. Not sure as that's not something I deal with. And I say that as someone who would only reluctantly buy Intel (CPU - no one better for network cards)

1 hour ago, _cjd_ said:

Losing the Intel for transcoding may not be worth the trade, especially looking at power efficiency.

Yeah I've been agonizing over that, but a few things are swaying me:

 

1) I've been warned I could run into "fun and interesting" performance characteristics with Intel's P and E-cores, which aren't always handled well in schedulers other than Windows (for now at least).

 

2)  The R9 7900 looks pretty power efficient   :

 

3) Adding a GPU doesn't seem like it will add that much to my idle power draw. According to SpaceInvaderOne it's possible to get a GTX 3080 down to ~8 watts on idle:

 

4) The R9-7900 is a much faster CPU.

 

My biggest hesitation is I assume the R9-7900 will have significantly higher idle power draw than a i5-14500 would. I can't find solid #s on that yet, but that will either solidify my decision or pull me back onto the fence.

 

-----------------------------------------------

 

EDIT: Found this thread which concludes:

Quote

"a System for Ryzen 7000 series float around 70~80w while some results showed they can go as high as 100w during idle, whereas Intel Alder Lake and Raptor Lake system could go as low as 60w."

 

They also mention:

Quote

"all 7000 series in this test were paired with a very high end power hungry X670 Extreme chipset"

🙁

 

Ugh I just don't know what to do. I wish Intel didn't have the microcode issues with their 13th/14th gen which limits me to the older/lower-end stuff if I want to avoid it. I wish the new Arrow Lake was more of a slam dunk regarding idle power efficiency. I wish AMD had better video transcoding. I wish ECC-capable motherboards were easier to find.

 

Sorry to vent. I've spent too much time obsessing over this and need sleep.

Edited October 31, 20241 yr by Gazeley

I can sympathize with the struggle. I run an asrock x570-d4u/5600x/32gb ECC and wonder if ECC and IPMI are worth keeping... idle for me is a mean of 46w, low of 38w. It used to be worse though (HBA card changed asm1166 expander, settings tweaks, fan curves the big improvements for me). I could lose fans at the expense of drive temps (still safe levels, maybe I'm too worried there) but the big issue is I can't idle to stopped fans.

Your Intel picks wouldn't idle as low as a different chipset, but then you couldn't ECC. 8w adds up across everything, but I bet with the AMD you can ditch the HBA so that'd be a power trade. Maybe you'd be fine with a b650 or whatever is the right one here - just will be fewer pcie lanes I think.

Skeptical on the power supply too - any documentation/independent testing? I've been burned so many times by PSU brands I don't recognize - long ago started being a lot more cautious.

My peak rate is $0.13 per kWh, so the difference between an idle 50W system and a 100W system is only about $5/mo for me. I guess then the question becomes ‘is the extra performance of the AMD worth $60/year to me?’

 

This is assuming the system is idling 100% of the time (which it won’t) and the R9-7900 does seem very efficient under load compared to Intel. Depending on how hard I’m using it there’s theoretically some point where it becomes more power efficient than an i5-14500 (although I doubt I’ll be using it hard enough to hit that point)

 

I also just noticed that the ASUS ProArt X670E-CREATOR has onboard 10Gb ethernet. Which means I could potentially go without the SFP+ NIC, except my switch only has SFP+ 10G ports and those 10GBase-T transceivers run HOT and presumably aren’t power efficient. So maybe I’d want to get the SFP+ NIC regardless so i can use a DAC.

8 hours ago, _cjd_ said:

Skeptical on the power supply too - any documentation/independent testing?

It was recommended to me over on level1tech forums. I've never heard of FSP either but it gets really good reviews.

Quote

There is a new king in the 1000W category. The FSP Hydro Ti Pro 1000W unit offers jaw-dropping performance in almost every area, especially transient response, which matters the most in real-life conditions. With the lowest voltage drop at 12V so far in the 200% transient response test, required by the ATX v3.0 standard in units equipped with 12VHPWR connectors, I can say that I am impressed, something that I rarely admit, after evaluating thousands of power supplies.

 

It's expensive (Titanium PSUs perform 3%-5% better in the 20W-80W range. So between a 0.6W - 1.6W difference. At best this would save about $5 a year vs any other 80+ Platinum) but I don’t mind paying a premium for good stuff. I’m hoping to get at least a decade of use out of this server - and I’m willing to splurge a bit for better build-quality/warranty/efficiency.

I see 2-3w power consumption on the switch from SFP+ to rj45 adapter when I fire up a computer with 10gbe (two different adapters, two different computers). Less than half a watt for similar but fiber DAC, and even less for copper DAC.

Thanks for the PSU links. Will see how they hold up over time, that's where I've had stuff go south.

With attention you can certainly tune AMD to be reasonable. My network stack (including server) as a whole is still a huge chunk of my annual power bill - not every day, but it's constant.

Ok the consensus here and on level1techs seems to be the following:

• Losing Intel's superior transcoding isn’t worth the trade up in overall performance with AMD
• The 14500 should be plenty good for my needs.
• 1000W PSU is overkill.
• Probably won’t need all 6 case fans.

 

I went ahead and changed the CPU back to the i5-14500, and the mobo back to the ASUS Pro WS W680-ACE (I want ECC). I also changed the PSU to a Seasonic Vertex PX-750 since I can’t find any good Platinum/Titanium options around 600W. I’m willing to splurge a bit on the PSU for top-notch build quality and efficiency.

 

The Seasonic Vertex PX-750 has 4 SATA power outputs - so I imagine with 4 of these CableMod cables I’ll be all good:

These guys have a pretty good list about different psu's efficiencies

https://www.cybenetics.com/index.php?option=whats-new

Supermicro MBD-X13SAE series might be an alternative if you're unsure about the Asus. More onboard SATA, but IMO the best pcie layout is the -F-O which brings IPMI (something I'd want, you may not). More $ though, and a PCI slot... Not sure if you wanted 2.5g onboard but that's another difference.

5 hours ago, _cjd_ said:

Supermicro MBD-X13SAE series might be an alternative if you're unsure about the Asus.

Ooooh, thanks for the suggestion! I like those extra SATA ports; with 8 on-board I could get away with a 9201-8i instead of a -16i. And I definitely trust Supermicro's build quality more than Asus. Looks like I'd have to pick out different memory though, I have this 4800 MHz DRAM on the build list but the specs of the X13SAE-F only show 4400/4000/3600 as supported speeds.

 

This is definitely my new top contender for the mobo though. Appreciate you bringing it to my attention!

You may want to read the data sheet for the ASUS board.

Quote: Total supports 3 x M.2 slots and 8 x SATA 6Gb/s ports // 1 x SlimSAS Slot Support SlimSAS NVMe device (supports PCIe 4.0 x4 mode and up to 4 SATA devices)

The SlimSAS port can be switched to SATA in the BIOS and you only need to buy the corresponding cable separately.

 

The only advantage of the Supermicro is that IPMI is directly on the board. However, the fan control is worse with the Supermicro because it only has two zones. Personally, I can recommend Asus. I am very happy with it and had Supermicro before.

Asus are also quicker with updates for BIOS and KVM. The latest microcode doesn't even seem to be available on the Supermicro website.

@enect Thank you for your comment. I didn't catch that note about the SlimSAS stuff. You've convinced me, I changed my mobo pick back to the W680-ACE.

 

 

The difference is how it reaches 8 ports - that may impact power consumption. Hard to know and expensive to validate. I'd agree either is going to be a solid choice - though supermicro may be in trouble? So perhaps not the better option anyway.

In the end, set your goals, commit, and don't look back.

Consider mini-pc options with a connected DAS. You will use much less power and should be more then capable for transcodes (INTEL ONLY).