UnRAID on Asus Pro WS W680-ACE IPMI

[demps]

8 hours ago, mrhanderson said:

 

Have you disabled Fast Boot?

If you have a 13th/14th gen Intel with iGPU, have you blacklisted the ast?

 

Without both of those I believe you are likely to experience boot/reboot issues.

 

I'll check fast boot. For blacklisting ast, assume I'm an idiot, what do you mean?

 

 

4 hours ago, theDrell said:

I probably need to plug the motherboard fans into the IPMI card, but I was short on Molex connectors to get the 6 pin PCIE to it, although I have a route.  But yeah. 

I am using a Noctua NH-D9L as my cpu cooler.

 

That cooler might not be up to the task of a 14900k but I would double check your mount. 

[Daniel15]

5 hours ago, sharpling said:

directly detected by unraid but cannot get 2_2 / 2_3 working no matter what i do.

Do you see the drives in the UEFI ("BIOS")? That'll at least narrow down if it's an issue with Unraid or if it's an issue with the hardware itself.

If you see them in the UEFI, then run `lsblk` in Unraid and see if you see them there. If you don't, try booting a Live CD of another linux distro (like a Ubuntu live CD) and see if you see the drives there.

 

I've got the mATX version which only has 2 M.2 slots, but I'm using both without issue (two M.2 drives in a ZFS mirror in Unraid)

 

6 hours ago, theDrell said:

 

 

I probably need to plug the motherboard fans into the IPMI card, but I was short on Molex connectors to get the 6 pin PCIE to it, although I have a route.  But yeah. 

I am using a Noctua NH-D9L as my cpu cooler.

Interesting... The NH-D9L moves about half as much air compared to the NH-D15, but that wouldn't explain such a large difference.

I'd recommend reapplying the thermal paste since my guess is that the CPU is making poor contact with the cooler. Remove the cooler, totally remove any existing thermal paste (using isopropyl alcohol if needed), then apply new thermal paste. If you're not sure how much to put or the pattern to use, watch Noctua's videos about it.

 

The other thing you could try is bumping the fan speed to 100% and see if it helps at all. Log in to the IPMI then go to settings → Fan control → Auto mode → Full speed mode. That'll force it to run at 100% all the time. I think 1600RPM is close to 100% speed for that fan though, so maybe it won't do anything.

Edited February 21 by Daniel15

[Neo78]

3 hours ago, Daniel15 said:

I've got the mATX version which only has 2 M.2 slots, but I'm using both without issue (two M.2 drives in a ZFS mirror in Unraid)

Hi Daniel,

 

may I ask you what are your power consumptions in idle at the moment?

[mikeyosm]

FWIW - here's mine, really happy with the results...

 

W680M-ACE-SE

14700

128GB DDR5

 

No VMs running

Approx 13 dockers running

 

Sits at around 36-60W idle depending on UNRAID process / dokcer usage.

Temps are pretty decent given I'm using the Thermalright Silver Soul 92mm cooler. 

 

  

 

[sharpling]

On 2/21/2024 at 7:47 AM, Daniel15 said:

Do you see the drives in the UEFI ("BIOS")? That'll at least narrow down if it's an issue with Unraid or if it's an issue with the hardware itself.

If you see them in the UEFI, then run `lsblk` in Unraid and see if you see them there. If you don't, try booting a Live CD of another linux distro (like a Ubuntu live CD) and see if you see the drives there.

 

I've got the mATX version which only has 2 M.2 slots, but I'm using both without issue (two M.2 drives in a ZFS mirror in Unraid)

Thank you,

Think I found the problem, 

The mobo seems to use port sharing between slimsas and pch nvme slots. I have slimsas connected to a sas backplane - disconnect it and the nvme drives are detected / working.

I will test if breakout cable allows any sata drives at all but i think it's the whole port - so no slimsas ports in case you want use pch m.2_2 or m.2_3.

Update: after putting back hba’s I now noticed that connecting slimsas also disables the x4 pcie slots so the problem points more towards bad cable or hardware.

Edited February 23 by sharpling

[mikeyosm]

29 minutes ago, sharpling said:

Thank you,

Think I found the problem, 

The mobo seems to use port sharing between slimsas and pch nvme slots. I have slimsas connected to a sas backplane - disconnect it and the nvme drives are detected / working.

I will test if breakout cable allows any sata drives at all but i think it's the whole port - so no slimsas ports in case you want use pch m.2_2 or m.2_3.

 

Oh no, hope not. I have the mATX variant and planned on using the slimSAS in SATA mode as well as both the m.2 slots. I guess we'll see as soon as I receive the cable.

[sharpling]

7 hours ago, mikeyosm said:

Oh no, hope not. I have the mATX variant and planned on using the slimSAS in SATA mode as well as both the m.2 slots. I guess we'll see as soon as I receive the cable.

Yeah, I also planned for a smaller 8 x sata array and have no room left for another HBA card.

Now also tried to disable the slimsas sata ports in bios individually and in groups. Thinking that maybe disabling 1-2 or 3-4 would open at least one m.2 but no diffrence. Hopefully it's backplane or cable related but i kinda doubt it. 

Anyways I opened a support case and asked for clarification related possible port sharing between PCH slimsas and PCH m.2 slots.
 

Update: after putting back hba’s I now noticed that connecting slimsas also disables the x4 pcie slots so the problem points more towards bad cable or hardware.

Edited February 21 by sharpling

[theDrell]

9 hours ago, Daniel15 said:

Interesting... The NH-D9L moves about half as much air compared to the NH-D15, but that wouldn't explain such a large difference.

Yeah Noctua claims it’s medium for even overclocking the 14900k. I turned all the fans to 100% last night and turned off some of the turbo boost for now. Temps are manageable now. 
 

 

I ordered a different Larger Noctua cooler and Amazon agreed to return this one. I’ve built pcs for over 20 years, so although I won’t say the thermal paste is perfect, it should have been good enough to not be maxing temps. 

 

I’ll swap to the bigger one and let you know. I’m building in a 4u Supermicro sc846 case so am limited some on cooler height. NH-D12L

Edited February 21 by theDrell

[mikeyosm]

6 minutes ago, theDrell said:

Yeah Noctua claims it’s medium for even overclocking the 14900k. I turned all the fans to 100% last night and turned off some of the turbo boost for now. Temps are manageable now. 
 

 

I ordered a different Larger Noctua cooler and Amazon agreed to return this one. I’ve built pcs for over 20 years, so although I won’t say the thermal paste is perfect, it should have been good enough to not be maxing temps. 

 

I’ll swap to the bigger one and let you know. I’m building in a 4u Supermicro sc846 case so am limited some on cooler height. 

Not sure if you've seen this but should help get temps down even more

https://forums.unraid.net/topic/145432-i7-13700k-and-asus-pro-ws-w680-ace-ipmi-optimizations/

[Daniel15]

3 hours ago, theDrell said:

I turned all the fans to 100% last night and turned off some of the turbo boost for now. Temps are manageable now. 

You could try limiting the power draw too (reduce the maximum wattage). I just saw this article this morning:

 

https://www.tomshardware.com/pc-components/cpus/is-your-intel-core-i9-13900k-crashing-in-games-your-motherboard-bios-settings-may-be-to-blame-other-high-end-intel-cpus-also-affected

 

There's two settings in the BIOS for it: One for max power draw normally, and one for max power draw during turbo.

 

 

Edited February 21 by Daniel15

[lishpy]

Has anyone been able to get fan control working with this board through unRaid and NOT IPMI (for those of us who have the non Ipmi boards)?

 

I'm not pleased with the drive temps I'm seeing and want to ramp up my fans a bit.

[mrhanderson]

On 2/20/2024 at 11:35 AM, Daniel15 said:

Won't this cause issues when trying to view the system via IPMI? There's a multi monitor setting in the BIOS that lets you use both the iGPU and the Aspeed GPU at the same time. 

 

I believe so. I have not done it myself yet (I'm still on 6.12.4 which was the last unaffected version). Another user that had experienced the same issues as me when trying to move to 6.12.6 reported that it had fixed the issue for them though. The multimonitor setting seems to have no impact on the issues that arise. Namely loss of iGPU access and reboot hangs. The only provided solution that seems to have worked so far is to blacklist the IPMI GPU.

 

On 2/20/2024 at 8:17 PM, demps said:

 

I'll check fast boot. For blacklisting ast, assume I'm an idiot, what do you mean?

 

See https://forums.unraid.net/topic/148775-unraid-os-version-6126-available/?do=findComment&comment=1346775

 

[Daniel15]

44 minutes ago, mrhanderson said:

 

I believe so. I have not done it myself yet (I'm still on 6.12.4 which was the last unaffected version). Another user that had experienced the same issues as me when trying to move to 6.12.6 reported that it had fixed the issue for them though. The multimonitor setting seems to have no impact on the issues that arise. Namely loss of iGPU access and reboot hangs. The only provided solution that seems to have worked so far is to blacklist the IPMI GPU.

 

 

See https://forums.unraid.net/topic/148775-unraid-os-version-6126-available/?do=findComment&comment=1346775

 

Ahh, interesting. I'm still on 6.12.4 because the SR-IOV plugin didn't support newer Unraid versions for a while. That's been fixed now, but I haven't had time to upgrade. I'd guess that some change in the Linux kernel between those two versions is causing the issue.

6.12.4 uses Linux kernel 6.1.49 whereas 6.12.6 uses Linux 6.1.64.

I don't see many changes to the ASpeed `ast` driver in the 6.1.x kernel branch: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/gpu/drm/ast?h=linux-6.1.y. I do see some changes in November 2023 in the latest dev branch, at least in Linus' kernel repo: https://github.com/torvalds/linux/commits/master/drivers/gpu/drm/ast. Those changes were for Linux 6.8 but I wonder if some of them have been backported into the kernel that Unraid uses. Of course, it could also be a change outside the ast driver that's causing the issues.

 

Has anyone reported this as a kernel bug yet?

Edited February 22 by Daniel15

[jakea333]

On 2/20/2024 at 1:35 PM, Daniel15 said:

Won't this cause issues when trying to view the system via IPMI? There's a multi monitor setting in the BIOS that lets you use both the iGPU and the Aspeed GPU at the same time. 

 

You'll lose the IPMI display while Unraid is booting. For me, there's not much to "see" there or interact with once Unraid is booted. My IPMI usage is focused on accessing the BIOS and verifying the boot process is starting correctly. I'm not using the main CLI via IPMI, so not much is lost.

 

2 hours ago, mrhanderson said:

 

I believe so. I have not done it myself yet (I'm still on 6.12.4 which was the last unaffected version). Another user that had experienced the same issues as me when trying to move to 6.12.6 reported that it had fixed the issue for them though. The multimonitor setting seems to have no impact on the issues that arise. Namely loss of iGPU access and reboot hangs. The only provided solution that seems to have worked so far is to blacklist the IPMI GPU.

 

 

I can confirm that it seems to fix the issue for me on 6.12.6. I've also upgraded to 6.12.8 with the same bugs and same fix still in place.

 

It seems this problem is limited to the mATX variant (I can see at least 3 confirmed cases through this thread). Doesn't seem the ATX version is reporting the same problem. Will be interested in seeing how @Daniel15 fares with your upgrade, as you seem to have a better understanding of this process than myself.

[Daniel15]

1 minute ago, jakea333 said:

For me, there's not much to "see" there or interact with once Unraid is booted

I sometimes use it to watch the boot process (make sure nothing fails during Unraid startup) and the shutdown process (e.g. see if VMs are not shutting down cleanly or filesystems aren't unmounting cleanly), so losing it isn't ideal. I'll try dig into it when I have time, but it's hard for me to get enough free time to do that.

[demps]

5 hours ago, jakea333 said:

It seems this problem is limited to the mATX variant (I can see at least 3 confirmed cases through this thread). Doesn't seem the ATX version is reporting the same problem. Will be interested in seeing how @Daniel15 fares with your upgrade, as you seem to have a better understanding of this process than myself.

 

I have the ATX version. Going to try your fix this weekend.

[sharpling]

Has anyone tested any nvme to nvme transfer speeds?

Since I am new to unraid I'm not sure how to proper test things but played around copying large files in 'mc' between two Corsair mp600 pro xt. Seems to max out at around 1.7GB/s, was expecting a bit more.

i7-13700, 64GB

[mikeyosm]

I got between 3 & 4 GB/s in mc copying vm flat files between 2 wd pice4. Not quite windows speeds but it'll do. 

[sharpling]

1 minute ago, mikeyosm said:

I got between 3 & 4 GB/s in mc copying vm flat files between 2 wd pice4. Not quite windows speeds but it'll do. 

Thank you, that is roughy what i was expecting to see.
I will play around a bit more, probably i've set something wrong somewhere.
 

[Daniel15]

23 minutes ago, sharpling said:

Thank you, that is roughy what i was expecting to see.
I will play around a bit more, probably i've set something wrong somewhere.
 

 

Which filesystem are you using?

[sharpling]

On 2/23/2024 at 8:02 PM, Daniel15 said:

 

Which filesystem are you using?

I've tested all filesystems now.

Even installed windows, crystal speed test aprox 7000MB/s.
Windows explorer file copy jumping between 3000-4000. 

Upgraded firmware on nvme's, reset mb bios, took out everything else from computer only left nvme's, reset unraid config. 

Messed around a bit copying without assigning them to a pool, not a big diffrence.

Made some new pools for the 11 time, changed to zfs (for 3rd time).
Then just suddenly stable 5GB/s between m2_2/m2_3 and 6GB/s if copying from m2_1.

I have absolutly no idea why.

Update: speed was due to beeing in cache. After a lot of trying i can get max 2GB/s copying from nvme to nvme so still suspecting something is off.
 

Edited February 26 by sharpling

[theDrell]

On 2/20/2024 at 11:47 PM, Daniel15 said:

Interesting... The NH-D9L moves about half as much air compared to the NH-D15, but that wouldn't explain such a large difference.

I'd recommend reapplying the thermal paste since my guess is that the CPU is making poor contact with the cooler. Remove the cooler, totally remove any existing thermal paste (using isopropyl alcohol if needed), then apply new thermal paste. If you're not sure how much to put or the pattern to use, watch Noctua's videos about it.

 

It wasn't my worst paste job, wasn't my best. Injury keeps me from really putting effort in right now.  I just swapped it to the NH-D12L and about to update some settings to see if the cooling is any better.

It is better, but reading some, the ASUS boards really just let whatever power into these 14900k as they can.  Manually setting power limits on it and decreasing the CPU current is keeping the processor temps down better.  I would like to underclock a hair as well.  Setting Pl1=PL2=125 and current to 307 which is what the current is specd to is maxing out at about 80 now with lots of cores running.  Going to keep an eye on it and try and figure out where exactly on this mobo to undervolt. 

Edited February 24 by theDrell
adding more info

[sharpling]

1 hour ago, theDrell said:

It wasn't my worst paste job, wasn't my best. Injury keeps me from really putting effort in right now.  I just swapped it to the NH-D12L and about to update some settings to see if the cooling is any better.

It is better, but reading some, the ASUS boards really just let whatever power into these 14900k as they can.  Manually setting power limits on it and decreasing the CPU current is keeping the processor temps down better.  I would like to underclock a hair as well.  Setting Pl1=PL2=125 and current to 307 which is what the current is specd to is maxing out at about 80 now with lots of cores running.  Going to keep an eye on it and try and figure out where exactly on this mobo to undervolt.

Is there a big diffrence between the max 90 degree setting and manually limiting?
 

[theDrell]

23 minutes ago, sharpling said:

Is there a big diffrence between the max 90 degree setting and manually limiting?
 

I have that disabled even now. Initially I had it set to the 90, but I would see it hit 100 sometimes.  Now the cooler has 50% more cooling headroom but it’s still hot. I had always planned to undervolt a little. I just went i9 cause I wanted a few more cores and maybe swap it with my i7 desktop if needed. But dang it’s hot. 

[Neo78]

On 2/21/2024 at 10:41 AM, mikeyosm said:

Sits at around 36-60W idle depending on UNRAID process / dokcer usage.

Temps are pretty decent given I'm using the Thermalright Silver Soul 92mm cooler. 

 

Hi,

 

thanks for the information!

 

Do you have any problems with IPMI, iGPU (Intel QuickSync), slim SAS Port, stability etc. ? I'm still unsure which mainboard I should buy:

- Asus Pro WS W680M-ACE SE

- Gigabyte B760M DS3H DDR4

 

The Asus does support ECC RAM what I think can be useful for the future. Other user reports that the mainboard does consume too much energy

The Gigabyte is completely different but maybe consumes less energy and easier to manage.

 

best regards,