LeetDonkey Posted April 11, 2024 Posted April 11, 2024 (edited) Hello, I am in the process of upgrading my Unraid system to the following setup: Supermicro X12SAE Intel Xeon 1290P 4 x 32GB ECC RAM. Since I had a running system already and had plenty of time to conduct a few tests, I decided to see if I could determine what kind of impact adding components would have to the power consumption. The PSU used for the following is a Seasonic X-400 Fanless GOLD. A single 120mm fan is attached to cool the CPU. Mouse, Keyboard and USB flash drive attached. Monitor attached via display port I am booting off a Ubuntu Live USB, and letting it sit idle in the console RAM is Kingston Premier KSM32ED8/32HC ECC Udimm 1 x RAM sticks ~15.0W without ethernet cable in ~15.8W with ethernet cable in ~14.0W with ethernet cable in and powertop --auto-tune ~11.6W powertop --auto-tune and everything removed (USB flash drive, ethernet cable, keyboard, mouse and display port monitor) 2 x RAM sticks ~17.4W with ethernet cable in ~14.8W with ethernet cable in and powertop --auto-tune 3 x RAM sticks ~17.4W with ethernet cable in ~14.8W with ethernet cable in and powertop --auto-tune 4 x RAM sticks ~17.8W with ethernet cable in ~15.3W with ethernet cable in and powertop --auto-tune According to powertop all above scenarios are able to reach C8 on CPU pkg 4 x RAM sticks + JMB585 PCIE controller ~19.5W with ethernet cable in ~17.8W with ethernet cable in and powertop --auto-tune With JMB585 CPU PKG is only able to reach C3 lspci -vvv confirms that ASPM is not supported on that card, so that makes sense So a few questions: Why does adding RAM make such a little impact on consumption? Is it because I need to have some activity to make it actually consume power? Secondly, I was under the impression that only being able to reach C3 on CPU pkg would have a bigger impact on the power consumption? If what I've measured is true, it barely makes any sense to specifically go for PCIE cards with ASPM support in order to save power. (9500-16i/9600-24i vs 9305-24i) The measurements are a bit difficult as it is a gold power supply and is being loaded with like 5% making the efficiency drop. This may also be why I am not seeing alot of change when adding for example RAM sticks. I tried limiting the CPU pkg to C3 in the bios without adding PCIE cards: 4 x RAM sticks ~20W with ethernet cable in ~18.0W with ethernet cable in and powertop --auto-tune Confirmed that CPU PKG does not go below C3, it seems barely worth the effort if this is correct.. Update 21-04-2024: Some more measurements: All theese are with 4 RAM sticks, ethernet cable in and powertop --auto-tune: Intel X710-DA2 no cable attached: CPU PCI-E: 19.1W C7 confirmed PCH: 18.4W C8 confirmed LSI 9306-24i no cables attached: CPU PCI-E: 32.7W C3 confirmed PCH: 31.7W C8 confirmed LSI 9306-24i(PCH) + X710-DA2(CPU) no cables attached: 34.7W C7 confirmed 1 x Lexar NM790(M2 slots are attached to PCH): 15.9W C8 confirmed 2 x Lexar NM790(M2 slots are attached to PCH): 15.9W C8 confirmed LSPCI confirms that the NVME is present LSI 9306-24i(PCH) + X710-DA2(CPU) no cables attached + 2 x NM790(M2 slots are attached to PCH): 34.7W C7 confirmed So, as long as I stick to PCH PCI-E slots, then it does not matter if the PCIE device supports ASPM Edited April 21, 2024 by LeetDonkey Clarification 1 Quote
_cjd_ Posted April 11, 2024 Posted April 11, 2024 How are you monitoring power consumption? It matters differently to different people. What does each watt cost you annually? Every budget balances differently. And for some it's a game - how low can you go... Quote
LeetDonkey Posted April 12, 2024 Author Posted April 12, 2024 7 hours ago, _cjd_ said: How are you monitoring power consumption? It matters differently to different people. What does each watt cost you annually? Every budget balances differently. And for some it's a game - how low can you go... I use a power meter on the primary side of the power supply. My point is, IF the difference between C3 and C8 is only 2-3 watts, then chasing PCIE devices with ASPM support for the sole purpose of the CPU PKG going beyond C3 is probably not worth it. But something else might be in play here, which is why I am asking. Quote
_cjd_ Posted April 12, 2024 Posted April 12, 2024 Power meter is probably accurate enough - a lot of folks use UPS stats which aren't. I don't know enough about this to say. I'd be curious if that changes more dramatically with attached drives (even spun down) or other add-in cards. HBA may reach higher states but also use more power up front. Given the improvements folks find going from C3 to C8 there must be more to it than you've captured so far. Also Unraid may not match Ubuntu... I shaved nearly 100w at home standby use chasing lots of low numbers like this... Though Unraid is in that total and ditching the HBA for ASM1166 SATA was more than a couple W. I'm also accepting IPMI cost and not changing to Intel because of the value + cost vs theoretical improvement. I am sticking with VLAN on a 10g/SFP+ network vs onboard Ethernet to save just 2-3w - I can't seem to disable the ports so probably missing some savings, but the switch is also lower use... I will also say I appreciate the numbers you're sharing because that's digging in with real data where I only had theoreticals before (ITX isn't enough for my setup and most ultra low use recommendations seem to be... Also very few ECC examples). Making me itch to spend more than I'd save in many many years just to experiment. Quote
LeetDonkey Posted April 12, 2024 Author Posted April 12, 2024 20 minutes ago, _cjd_ said: Power meter is probably accurate enough - a lot of folks use UPS stats which aren't. I don't know enough about this to say. I'd be curious if that changes more dramatically with attached drives (even spun down) or other add-in cards. HBA may reach higher states but also use more power up front. Given the improvements folks find going from C3 to C8 there must be more to it than you've captured so far. Also Unraid may not match Ubuntu... I shaved nearly 100w at home standby use chasing lots of low numbers like this... Though Unraid is in that total and ditching the HBA for ASM1166 SATA was more than a couple W. I'm also accepting IPMI cost and not changing to Intel because of the value + cost vs theoretical improvement. I am sticking with VLAN on a 10g/SFP+ network vs onboard Ethernet to save just 2-3w - I can't seem to disable the ports so probably missing some savings, but the switch is also lower use... I will also say I appreciate the numbers you're sharing because that's digging in with real data where I only had theoreticals before (ITX isn't enough for my setup and most ultra low use recommendations seem to be... Also very few ECC examples). Making me itch to spend more than I'd save in many many years just to experiment. I believe it is accurate, I also have some smart plugs I could use for comparison, but this energy meter is somewhat expensive compared to smart plugs, and also endorsed by the national power companies, so I would assume it has some accuracy. In my Unraid server I have both APC smart UPS which is reasonably accurate and a Corsair AX760i which is definately not accurate. But I would like to wait until I have a good sense of what hardware to use before taking it apart. I am actually considering the switch you made, currently I have a 9305-24i(Actually a 9306-24i but it's the same hardware, power consumption wise) to ASM1166. I have one ASM1166 coming in soon to get some indication of what kind of power figures I would be looking at. On the downside I would need 4 x ASM1166 combined with a carrier board with some kind of PCIE packet switch as my board does not support Bifurcation. The packet switch will probably use a decent amount of power, so I would probably end up with something that consumes about the same as my current controller. The only upside would be that it would support ASPM. What kind of controller did you have before? And how much did it reduce your consumption to use ASM1166? The second thing I considers was a 9500-16i that supports ASPM and either replace some 4 TB drives for a 20TB to bring the sata interface need down to 16, or add an ASM1166 as an extra PCIE controller, giving me a total of 22 drives where I am currently in need of 19. But it all depends on the impact it will have on the total consumption. I have jumpers to disable onboard audio and ethernet ports, it did not make much of an impact, maybe 2W at most, I think the ethernet ports already go into low power mode when no cable is attached, so not a big impact, but everything counts. Quote
_cjd_ Posted April 13, 2024 Posted April 13, 2024 Power read is probably good enough - a point of comparison (via smart plug) might be interesting, regardless. I've got a Shelly on mine. It's quite a bit more sensitive than the clamp reading I have for the whole circuit in my load panel. I have an LSI9211-8i HBA. It was 10-12w total drop going to an ASM1166 m.2 adapter instead (10 sata drives in my system; 5 ssd, 5 spinners, only the latter were ever on the hba). At the moment power costs me ~us$1/yr/1w all in, so that's still a 2-3 year payback... with your *current* setup I'd be hard pressed to justify the hardware costs trying to get that many 1166; jumping to larger hard drives seems a better investment - the x4 and x1 slots could get you 8 sata ports comfortably; another 12 in the m.2 if you aren't planning nvme drives. But there IS a point of diminishing returns vs a good HBA with ASPM support I suspect. And not all ASM1166 seem to deliver good results. Of course I'm now thinking of a switch to a pcie card to open up the m.2 for a mirrored appdata setup (solo nvme drive for that right now) but... meh. backups and backups hopefully enough to get me running quickly enough in event of a failure. 19 drives are going to add quite a bit more idle power consumption (or 22) vs... fewer higher capacity drives too. Not to mention when they're all spun up, again making it a likely better investment if you are trying to reduce power consumption. Also the power supply demand for that many drives; fewer opens the door to more efficient power supply options (ironically, my sim rig has the best low-power supply I own... seasonic 750w titanium - but waiting to see what video card options open up in the future before buying a replacement to swap stuff around - and I have a multi-decade long grudge against nvidia so who knows when that'll happen) The downside to drive upsizing is parity has to happen first (and I hope with that many drives you've got dual parity in the mix) so the initial curve to larger capacity is $$. Quote
Vr2Io Posted April 13, 2024 Posted April 13, 2024 (edited) On 4/11/2024 at 6:42 PM, LeetDonkey said: Why does adding RAM make such a little impact on consumption? Is it because I need to have some activity to make it actually consume power? RAM really doesn't use lot of power, you would found even single phase power circuit would be enough, but if you add some voltage to it, then power will raise dramatically but still neglectable if compare to CPU power usage. For CPU, it also same, the pkg power was low because CPU run in minimal working voltage. I never concern deep C-states, I usually focus on those power hungry appliances, i.e. air-conditioner, heater etc. Take over control its cool on / off by automation have saving more. Edited April 13, 2024 by Vr2Io Quote
LeetDonkey Posted April 21, 2024 Author Posted April 21, 2024 I have updated the original post with some new measurements Quote
firstTimer Posted August 27, 2024 Posted August 27, 2024 (edited) On 4/11/2024 at 12:42 PM, LeetDonkey said: Hello, I am in the process of upgrading my Unraid system to the following setup: Supermicro X12SAE Intel Xeon 1290P 4 x 32GB ECC RAM. Since I had a running system already and had plenty of time to conduct a few tests, I decided to see if I could determine what kind of impact adding components would have to the power consumption. The PSU used for the following is a Seasonic X-400 Fanless GOLD. A single 120mm fan is attached to cool the CPU. Mouse, Keyboard and USB flash drive attached. Monitor attached via display port I am booting off a Ubuntu Live USB, and letting it sit idle in the console RAM is Kingston Premier KSM32ED8/32HC ECC Udimm 1 x RAM sticks ~15.0W without ethernet cable in ~15.8W with ethernet cable in ~14.0W with ethernet cable in and powertop --auto-tune ~11.6W powertop --auto-tune and everything removed (USB flash drive, ethernet cable, keyboard, mouse and display port monitor) 2 x RAM sticks ~17.4W with ethernet cable in ~14.8W with ethernet cable in and powertop --auto-tune 3 x RAM sticks ~17.4W with ethernet cable in ~14.8W with ethernet cable in and powertop --auto-tune 4 x RAM sticks ~17.8W with ethernet cable in ~15.3W with ethernet cable in and powertop --auto-tune According to powertop all above scenarios are able to reach C8 on CPU pkg 4 x RAM sticks + JMB585 PCIE controller ~19.5W with ethernet cable in ~17.8W with ethernet cable in and powertop --auto-tune With JMB585 CPU PKG is only able to reach C3 lspci -vvv confirms that ASPM is not supported on that card, so that makes sense So a few questions: Why does adding RAM make such a little impact on consumption? Is it because I need to have some activity to make it actually consume power? Secondly, I was under the impression that only being able to reach C3 on CPU pkg would have a bigger impact on the power consumption? If what I've measured is true, it barely makes any sense to specifically go for PCIE cards with ASPM support in order to save power. (9500-16i/9600-24i vs 9305-24i) The measurements are a bit difficult as it is a gold power supply and is being loaded with like 5% making the efficiency drop. This may also be why I am not seeing alot of change when adding for example RAM sticks. I tried limiting the CPU pkg to C3 in the bios without adding PCIE cards: 4 x RAM sticks ~20W with ethernet cable in ~18.0W with ethernet cable in and powertop --auto-tune Confirmed that CPU PKG does not go below C3, it seems barely worth the effort if this is correct.. Update 21-04-2024: Some more measurements: All theese are with 4 RAM sticks, ethernet cable in and powertop --auto-tune: Intel X710-DA2 no cable attached: CPU PCI-E: 19.1W C7 confirmed PCH: 18.4W C8 confirmed LSI 9306-24i no cables attached: CPU PCI-E: 32.7W C3 confirmed PCH: 31.7W C8 confirmed LSI 9306-24i(PCH) + X710-DA2(CPU) no cables attached: 34.7W C7 confirmed 1 x Lexar NM790(M2 slots are attached to PCH): 15.9W C8 confirmed 2 x Lexar NM790(M2 slots are attached to PCH): 15.9W C8 confirmed LSPCI confirms that the NVME is present LSI 9306-24i(PCH) + X710-DA2(CPU) no cables attached + 2 x NM790(M2 slots are attached to PCH): 34.7W C7 confirmed So, as long as I stick to PCH PCI-E slots, then it does not matter if the PCIE device supports ASPM Interesting topic and thanks for the contribution FYI, reaching C6 is usually the best compromise --> check this page out where this guy confirms that C6 to C8 difference is 1W only https://mattgadient.com/7-watts-idle-on-intel-12th-13th-gen-the-foundation-for-building-a-low-power-server-nas/ One question though for your second session of tests. (just to be the devil's advocate here) Did you remove the C-state limit of maximum C3 from the BIOS, right? There are some strange things here: As far as I know only HBA 9500 and 9600 support ASPM and 9400 can (with some issues though) Therefore with you HBA you should AT BEST reach C3 (disregarding if the slot is connected to PCH or CPU) So either powertop is bugged (presenting lower C-states than the device in reality is) OR motherboard is presenting some wrong values to powertop Following your tests though, it seems to me that, this "rush" (and time spent) to find the components with ASPM support to reach lower C-States does not really apply to every platform (I mean combination of CPU and Chipset) Edited August 27, 2024 by firstTimer Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.