Intel Socket 1151 Motherboards with IPMI AND Support for iGPU

[ionred]

9 hours ago, kaiguy said:

Thanks for the heads up, @Hoopster. I've never ran into that before, but I have had an intermittent issue where trying to login to the HTML5 IPMI gui would just keep going back to the login screen. The only way to correct it was to shut down and pull the power from the server, but I might try reflashing the BMC next time I encounter it.

If this happens, you can cold reset the BMC from the command line from your machine. I'm pretty sure it's 

ipmitool mc reset cold

but I could be wrong on that. I've had hte same issue you described with this board before with the BMC login loop and the BMC just needed a reset. Never lost any uptime on the machine itself.

[Burizado]

Yeah, I have a similar issue where the IPMI would just keep going back to the login screen as well.  I thought it was something odd in my network setup, but since nothing else seemed to be effected it wasn't high on my priority list to track down.

 

Sounds like this might be the same as what you both listyed above.  I will have to try reflashing next time I take the server down.  Thanks for the heads up!

[Burizado]

8 hours ago, ionred said:

If this happens, you can cold reset the BMC from the command line from your machine. I'm pretty sure it's 

ipmitool mc reset cold

but I could be wrong on that. I've had hte same issue you described with this board before with the BMC login loop and the BMC just needed a reset. Never lost any uptime on the machine itself.

@ionred would you do this from the Unraid terminal session?  Also, you mention this can be done while the server is running with no downtime?

[Hoopster]

5 hours ago, Burizado said:

@ionred would you do this from the Unraid terminal session?  Also, you mention this can be done while the server is running with no downtime?

ipmitool is not installed by default so if you need ipmitool, install it through NerdTools.

 

I did a few things with the BMC with ipmitool but for some reason it was not working directly from a terminal session on the target  machine.  I had to do it remotely with a terminal session from my other Unraid server. This requires the following syntax:

 

ipmitool -I lan -H <ip address of target BMC> -U admin -P <admin pw> <ipmitool commands>

Edited March 15 by Hoopster

[Burizado]

1 hour ago, Hoopster said:

ipmitool is not installed by default so if you need ipmitool, install it through NerdTools.

 

I did a few things with the BMC with ipmitool but for some reason it was not working directly from a terminal session on the target  machine.  I had to do it remotely with a terminal session from my other Unraid server. This requires the following syntax:

 

ipmitool -I lan -H <ip address of target BMC> -U admin -P <admin pw> <ipmitool commands>

@Hoopster Thanks for the info!  I don't have a test Unraid server, but I could load up one of my RasPis and probably do it from there, if it doesn't work directly on my server.  I'll give it a try once I have my server in a quiet time, in case something goes horribly wrong. 😅

Edited March 15 by Burizado

[ionred]

13 hours ago, Burizado said:

@ionred would you do this from the Unraid terminal session?  Also, you mention this can be done while the server is running with no downtime?

Yes this can be accomplished from the unraid terminal, either via web or via ssh to unraid. 
 

Fortunately I did not have to do it from a remote machine like Hoopster, it just “worked”. The BMC takes a couple minutes to fully start back up. By rough memory I’d say it was another 4-5 minutes before I could get to the login screen, but it let me login immediately afterwards without any ugly loops. The unraid system (and ssh session) stayed up the entire time. 

[JimmyGerms]

@Hoopster or anyone here do you happen to use bonding mode 802.3ad? I recently upgraded all my equipment and have a unifi 8 port switch with LAG and have set it up based on this guide but I'm always greeted with this message in the logs:

 

bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

 

I'm having a hell of a time setting up VLANs as well with dockers but slowly getting them to work. Only one that's really giving me trouble is CUPs but I think that's a docker image issue as it only works on the local host (bridge) and refuses to function correctly on vlans.

[Hoopster]

9 minutes ago, JimmyGerms said:

do you happen to use bonding mode 802.3ad?

I do not.  The motherboard came with IPMI LAN and LAN2 bonded together and I have not attempted to break that to bond LAN1 and LAN2.

[Gonzo-the-great]

12 minutes ago, JimmyGerms said:

@Hoopster or anyone here do you happen to use bonding mode 802.3ad? I recently upgraded all my equipment and have a unifi 8 port switch with LAG and have set it up based on this guide but I'm always greeted with this message in the logs:

 

I have that configuration. I have the US-24, but LAG is the same across "almost" their entire switch line.

 

 

 

My config is on VLAN 10

The only message I get is from "Fix Common Errors" because of a complex bonding which I ignore. 

 

It's been flawless since it's been set up.

 

12 minutes ago, JimmyGerms said:

bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

 

I'm having a hell of a time setting up VLANs as well with dockers but slowly getting them to work. Only one that's really giving me trouble is CUPs but I think that's a docker image issue as it only works on the local host (bridge) and refuses to function correctly on vlans.

 

[JimmyGerms]

@Gonzo-the-great yup, that's exactly how mine is setup. Would you mind checking your syslogs for that warning? I wonder if it's a false positive?

[Gonzo-the-great]

Yup, I see that once in the log. About the time my system was rebooted, so quite possible that the team hasn't been fully set up yet.

[JimmyGerms]

Oh my gosh, I just realized in my IPMI console they are bonded...would this be an issue if I have aggregation enabled through unraid?

Edited March 19 by JimmyGerms

[Gonzo-the-great]

That bonding makes it so the 2nd adapter is a backup and only becomes active if the primary fails, so unRaid only sees one adapter. There are a number of bonding situations that do not involve a switch. Only 802.3ad will cause issues if you don't have a switch that supports it.

 

So, the short answer is NO, this will NOT cause issues.

Edited March 20 by Gonzo-the-great

[JimmyGerms]

Just a heads up, I had to disable the bonding in IPMI due to my unifi switch getting confused. My local IPMI address would become unavailable due to the link aggregation and bond. Seems to be ok once disabled...and I turned off DNS just to see what would happen. Everything seems fine after a week!

[int2e]

BIOS version 2.40 will be updated on May 16, 2022, and the system is down today

Edited April 4 by int2e
wrong description

[Shane01638]

I'm still on L2.10A, is there a reason to upgrade to 2.40?  I wasn't going to upgrade until the turbo issue was fixed.

[Hoopster]

7 minutes ago, Shane01638 said:

I wasn't going to upgrade until the turbo issue was fixed.

I doubt the Turbo Boost issue will ever be fixed. Socket 1151 boards are fairly long in the tooth now (in computer terms). I am not sure ASRock even recognizes it as an issue.

 

I am on BIOS L2.35 and it still exists there.

[int2e]

9 hours ago, Shane01638 said:

我还在使用 L2.10A，有升级到 2.40 的理由吗？在涡轮问题得到解决之前，我不会升级。

no

[int2e]

 

20 hours ago, Hoopster said:

I doubt the Turbo Boost issue will ever be fixed. Socket 1151 boards are fairly long in the tooth now (in computer terms). I am not sure ASRock even recognizes it as an issue.

 

I am on BIOS L2.35 and it still exists there.

 

I am on BIOS L2.40 and it still exists there.

[Hoopster]

PSA for those who have a case that limits CPU cooler height.  I recently discovered the Noctua NH-D12L CPU cooler which has a 145mm height and fits many 4U cases.  Up to this point, the best I could do was the Noctua NH-U9S with a 125mm height and 92mm fans. 

 

My Silverstone CS380 case has a cooler height limit of 146mm.

 

I installed the NH-D12L on my E3C246D4U/E-2288G system and saw immediate results. Idle temps are 3-5C lower and under heavy load, CPU temps are up to 15C lower than with the NH-U9S cooler.  That 120mm fan makes a difference. 😀

 

At US$90, it is an expensive CPU cooler but the results are worth it for me.

Edited April 28 by Hoopster

[Hoopster]

Here is some information from Noctua that helps explain the difference in temps noted in the prior post:

 

The NH-D12L is rated best for overclocking/turbo with the E-2288G (arrow to the far right) as seen below

 

The NH-U9S is rated for moderate overclocking/turbo with the E-2288G (arrow in the middle) as seen below

 

In the NSPR scale Noctua uses to rate cooler performance (rather than a TDP rating), the NH-D12L scores a very respectable 148 and the NH-U9S is rated much lower at 93.

[ServerUngeek]

Does anybody here have Asrock E3C246D4U or variants (I have E3C246D4U-2L2T) and can confirm to me that if you disable BMC by means of the corresponding jumper BMC_DIS1 than the bmc seems to be disabled however the board CONTINUES to draw 8Watt even if the computer is shut off??? This is absolutely not the case with other boards where you can disable BMC. Or am I missing the point?

If this is standard behaviour, It's by far the worst and sloppiest possible design I can imagine. Asrock, are you kidding me?

Also, the maximum package C state I am able to reach with this board and Xeon-2144G is C3. What a terrible board.

Edited May 5 by ServerUngeek