UnRAID on Asus Pro WS W680-ACE IPMI

[Lolight]

7 hours ago, golfsands7 said:

Not that I disagree if power consumption is the primary objective but apparently STH got their X13SAE to idle at 31W. It leads me to wonder if the difference being observed here is partly due to the configurations as well. 

Sure, the point being that it's much easier to achieve good idle efficiency on the simple consumer boards.

[Neo78]

31 minutes ago, Lolight said:

Sure, the point being that it's much easier to achieve good idle efficiency on the simple consumer boards.

I agree ... I'm a bit skeptical about the stability of consumer boards for 24/7 usage, the W680 board should be the better, more secure option for this. I think I'll go with the WS W680M-ACE SE, even if it requires more energy

[Lolight]

18 minutes ago, Neo78 said:

I agree ... I'm a bit skeptical about the stability of consumer boards for 24/7 usage, the W680 board should be the better, more secure option for this. I think I'll go with the WS W680M-ACE SE, even if it requires more energy

The vast majority of consumer boards with maintained BIOS support (Asus, Asrock, Gigabyte, MSI etc) are designed and capable of securely running 24/7.

There's absolutely no advantage in choosing the W680 based boards over others, unless you require their unique features.

Features that are not available in the consumer chipset boards.

And If you don't need/use those then you're overpaying for increased complexity and power consumption.

Edited January 9 by Lolight

[Neo78]

26 minutes ago, Lolight said:

There's absolutely no advantage in choosing the W680 based boards over others, unless you require their unique features.

Features that are not available in the consumer chipset boards.

And If you don't need/use those then you're overpaying for increased complexity and power consumption.

thanks for the input. Another feature of W680 boards maybe is ECC support for unraid (ZFS). Can I ask which mainboard  you are using? My alternative as a consumer board would be "Gigabyte B760M DS3H DDR4"

[Lolight]

44 minutes ago, Neo78 said:

Another feature of W680 boards maybe is ECC support for unraid (ZFS). Can I ask which mainboard  you are using? My alternative as a consumer board would be "Gigabyte B760M DS3H DDR4"

Sure, ECC compatibility if desired will force you into the W680.

I've been running an old Asus B250 - it has all I need.

[Daniel15]

2 hours ago, Lolight said:

ECC compatibility if desired will force you into the W680.

Only because Intel artificially limit ECC to workstation and server boards. There's no technical reason ECC couldn't work on consumer boards given the memory controller is on-die. Thanks, Intel.

[Lolight]

30 minutes ago, Daniel15 said:

Only because Intel artificially limit ECC to workstation and server boards. There's no technical reason ECC couldn't work on consumer boards given the memory controller is on-die. Thanks, Intel.

Yeah, Intel have artificially removed ECC compatibility from the non-business sector of the market.

Therefore making its implementation far more costly than it would be otherwise.

 

But IMO, ECC RAM is not critical and doesn't need to be used in a home NAS environment.

The decision to stick to it makes it a personal choice.

 

That's all I'm going to say.... (not here to start a war over ECC and its place in a home NAS).

[Daniel15]

On 1/9/2024 at 11:52 AM, Lolight said:

But IMO, ECC RAM is not critical and doesn't need to be used in a home NAS environment.

 

In my case, my NAS stores backups from production web and email servers as well as important photos and documents (tax documents, title deed for house and car, etc) so data integrity is important. I don't store many 'disposable' files (TV shows, etc) on it. I take daily backups using Borgbackup (and retain one backup per month indefinitely) but there's a risk that silent corruption can also result in backups getting corrupted too. 

 

Totally understand that some people are happy not using ECC, and that's fine too. 

Edited January 10 by Daniel15

[casperse]

On 1/8/2024 at 11:14 AM, NAS-newbie said:

I have 63W (at 7% with a T-series i9 CPU) but then I run two enterprise SSD for cache (U.2 NVMe), 3xPCIe sticks (idle at this point) and 4x20TB WD Enterprise HDDs (powered down right now). No GPU as I rely on the one in the CPU for my plex needs.

 

I am also trying to get the power consumption down on my i9 but the latest Bios update (3101) have a new setting for E-cores?
Do you? or anyone here know what settings is recommended to use here? (try to follow the guide below):

Also I would like to know what U.2 NVMe drives are you using and what adapter?

I have seen these (Intel DC P4510 Series SSDPE2KX080T801 8TB) go down in price lately even used ones are worth it since they last a very very long time.
I gues it could be running in a 4x PCIe slot?
And I could scrap my Nvme adapter on the 8x slot 🙂

[firstTimer]

On 1/8/2024 at 10:51 AM, FirbyKirby said:

Good point, I do indeed:

• 3x M.2 NVMe
• 1x NVIDIA GTX 4060
• 1x Mellanox 2-port QSFP+ (with DAC for 10 GbE.)

Additionally, though they're not spun up, I've got 5x SATA drives (3x HDD and 2x SSD).

Are you mounting you GPU inside a VM? If yes, keep the VM turned on, so the OS on the VM will lower the GPU state to the miminum, otherwise if you stop the VM the gpu will run with higher P-State since Unraid can't control it though Nvidia Drivers

[FirbyKirby]

3 hours ago, firstTimer said:

Are you mounting you GPU inside a VM? If yes, keep the VM turned on, so the OS on the VM will lower the GPU state to the minimum, otherwise if you stop the VM the gpu will run with higher P-State since Unraid can't control it though Nvidia Drivers

Oh! Thank you. That explains a ton. I haven't setup pass-through yet, so the GPU is just sitting idle in Unraid. I'll be very curious to see how that impacts the power usage.

[firstTimer]

Anyone has run powertop when running Unraid? The max (lowest c-state I get is C3)

[Daniel15]

3 hours ago, firstTimer said:

Anyone has run powertop when running Unraid? The max (lowest c-state I get is C3)

 

I don't reach any pkg C-states and haven't had time to figure out why

 

What CPU do you have, and what PCIe devices do you have in the system? Some PCIe devices prevent the CPU from reaching deeper states than C3. You could try booting from a live CD with all PCIe cards removed and see if that helps. 

[firstTimer]

2 hours ago, Daniel15 said:

 

I don't reach any pkg C-states and haven't had time to figure out why

 

What CPU do you have, and what PCIe devices do you have in the system? Some PCIe devices prevent the CPU from reaching deeper states than C3. You could try booting from a live CD with all PCIe cards removed and see if that helps. 

The cpu is an intel i5-13400 and as PCIe devices I have the included IPMI card and a gtx 1050 (passed to a windows 11 VM). Anyway I don't reach any C-states in the Pkg as well and I thought it was the Powertop software not updated but I have the latest 2.15. Today though, 1 thing allowed the cpu to go to C3 state, it was after I removed the only sata SSD I have (Silicon Power that I have in signature). To be honest though, there wasn't sufficient difference in watt consumption (just a couple watt less) to justify the purchase of another ssd 😁Therefore, my suspicion now goes to the IPMI card now, or the on-board dual nic 🤔 but... in that case, I think I'll have to live with it.

Edited January 20 by firstTimer

[firstTimer]

@Daniel15 I think a guy found out something interesting, it seems like Intel 13th gen and 14th are not "fully supported" that's why maybe also powertop is not helping us https://forums.unraid.net/topic/151632-unraid-power-inefficiency-on-14-series-intel-processors/#comment-1357238

[golfsands7]

On 1/20/2024 at 4:27 PM, firstTimer said:

The cpu is an intel i5-13400 and as PCIe devices I have the included IPMI card and a gtx 1050 (passed to a windows 11 VM). Anyway I don't reach any C-states in the Pkg as well and I thought it was the Powertop software not updated but I have the latest 2.15. Today though, 1 thing allowed the cpu to go to C3 state, it was after I removed the only sata SSD I have (Silicon Power that I have in signature). To be honest though, there wasn't sufficient difference in watt consumption (just a couple watt less) to justify the purchase of another ssd 😁Therefore, my suspicion now goes to the IPMI card now, or the on-board dual nic 🤔 but... in that case, I think I'll have to live with it.

Dont know if linking is allowed but on STH someone said that the IPMI card didn't allow for deeper C states

[firstTimer]

13 minutes ago, golfsands7 said:

Dont know if linking is allowed but on STH someone said that the IPMI card didn't allow for deeper C states

Mmm... I actually suspected that. It would be nice if someone could confirm this, or otherwise I will test it myself when I have some time. So it means that buying this MB was a waste (for a c-state conscius user) because me and maybe other users chose this MB for the "seamless" IPMI card integration. Other features like 3xNVME is supported by other MBs as well (which cost less).

Edited January 22 by firstTimer

[Daniel15]

1 hour ago, firstTimer said:

@Daniel15 I think a guy found out something interesting, it seems like Intel 13th gen and 14th are not "fully supported" that's why maybe also powertop is not helping us https://forums.unraid.net/topic/151632-unraid-power-inefficiency-on-14-series-intel-processors/#comment-1357238

 

The powertop Github issue that post links to is one that I created. I only recently looked at the code. I still haven't heard back from the maintainer of the repo (who as far as I know still works at Intel). I was going to email the maintainer of Intel_idle to ask them about it too. 

 

[firstTimer]

Do you guys have the same issue? every time I try to set Native ASPM to enabled, when I reboot and go inside bios again, it is always set to Auto. I even tried to set Disabled but when I come back the next reboot, it is set back to Auto, any clue?

I am latest bios 3101

 

[SpinCykle]

Good day gentlemen!

 

I recently built a new server with this board (12600k, 64 gb  ddr5 4800 ecc mems, LSI 9300 16i) and I’ve been getting post code FF after booting into unRaid. I’m on the latest BIOS/Intel ME firmware. Is anyone else getting this q code error. I've removed all none essential hardware (LSI card, IPMI card, ect.) and FF is always displayed after boot/POST to unRaid. The logs are clean and the system appears to be functioning correctly.....

 

Thoughts?

Edited January 25 by SpinCykle

[demps]

Sorry if this was answered in this thread already but what is the trick to getting iGPU transcode working with this board? I'm assuming I'm need to change something in BIOS. 

 

I can plug a monitor in and get output but I can't get the iGPU to show up. /dev/dri only shows card0 in it. 

[Ben4]

I had to return this board as it was unstable for me. Every time I rebooted unraid, the server would not boot back up and kept power cycling off and on. Then I would have to unplugg the power cable for about 30 seconds and plug it back in to boot. Also the IPMI BMC did not work for me correctly, I can log in and launch the viewer to control the PC, but the moment I used BMC to reboot the machine, it would freeze and loose connectivity and do them same thing I mentioned before. The BIOS and the ME were updated to latest version. Asus offered to RMA the board or return it to Amazon for a refund, which I choose the latter.

 

I say this to ask if anyone has experienced the same issue? I'm wondering if I simply got a bad board and try again with a new one or if this particular board is bad all together and should I go with a different board?

 

Thanks for the input.

Edited January 28 by Ben4

[Ben4]

On 1/25/2024 at 2:16 AM, SpinCykle said:

Good day gentlemen!

 

I recently built a new server with this board (12600k, 64 gb  ddr5 4800 ecc mems, LSI 9300 16i) and I’ve been getting post code FF after booting into unRaid. I’m on the latest BIOS/Intel ME firmware. Is anyone else getting this q code error. I've removed all none essential hardware (LSI card, IPMI card, ect.) and FF is always displayed after boot/POST to unRaid. The logs are clean and the system appears to be functioning correctly.....

 

Thoughts?

If I remember correctly from the manual that came with the board, FF is a reserved code for future use. The codes are in the back of the manual.

[user2579]

On 9/18/2023 at 8:39 AM, frodr said:

I have this board, and I guess I'm the unlucky duck here. I had all kinds of trouble:

• Startup sequence randomly stuck on F6, specially after adding/removing pcie cards.
• Startup sequence randomly stays on OData Server info screen forever, or fully stuck.
• IPMI card looses contact with mainboard, sometime I have to physically remove/install it to get it up and running. 
• IPMI remote connection fall out "all the time".
• IPMI remote control due not follow the startup procedure until end. 
• Bios resets itself to default, mostly after tuning main power off. 

I'm in contact with Asus Support, that is sometimes like talking to a dement person asking the same question over and over again. 

 

A question to this board. Is the 2 pcie5 slots "connected? My understanding is the if I run 16 lanes card in slot 1, there is only 4 lanes to the CPU in pcie slot2? Does this mean a 8 lane card in pcie slot will not work? Or will it work with 4 lanes bandwidth?

 

// 

I have been going made on this as well, the F6 issue that requires a CMOS reset to clear as well as that OData Server issue.

 

My latest issue is it looks like the ECC RAM I have is either going bad or there's some kind of other issue as I'm getting all kinds of segfaults and process tainted issues.

 

You ever figure out the F6 issue?

[JBoY]

Hello, I am looking for buffered/registered ECC memories for the Asus Pro WS680M ACE SE with an i7-14700K. Do you know if they are compatible or this MB only accepts unbuffered? Even to indicate the maximum speed of 4800 MT/s it only appears in an image on the Asus website. It seems that there is not much information about the specifications in the pdf manual on the website itself.

I've seen that you have named the Kingstons (KSM48E40BD8KM-32HM and KSM48E40BS8KM-32HM)  but from what I understand they are non-buffered.

Thank you very much in advance.