Intel Socket 1151 Motherboards with IPMI AND Support for iGPU


Recommended Posts

Welp, after a couple days of success, it looks like I got another hard lock. I setup a syslog server (back to unraid) so I think I was able to capture some stuff, but I don't really know how to read this aside from noticing that I have a bunch of kernel warnings about the CPU being tainted. Anyone here have more experience reading this type of logging? Looks like something similar happened earlier in the day (but the server was still up), then it happened again and took everything down. I think.

 

https://pastebin.com/raw/afA6Wd5a

 

I have since disabled turbo boost. I should probably run a memtest (even though its ECC), but I think that means I need to disable EFI boot as when I try at bootup it just does a reboot. Ugh. Everything was running so smoothly for me before updating to 6.9.0/6.9.1...

 

Edit: Well, disabling turbo boost seems to have helped--no system locks since. Crazy how this was something others experienced with older BIOS versions and I never did. Then with the upgrade to 6.9.x, I'm suddenly getting it. It probably has nothing to do with it, but I decided to remove the graphite thermal pad I was using and replaced it with Arctic MX-5 compound. Going to run the system a few more days with turbo boost disabled then try re-enabling it. My thought process is one of the cores may have not been getting enough contact with the pad and that may have been causing my lockups. Complete guess, but I'm going with it.

Edited by kaiguy
New info
Link to comment

Server upgrades went well! The new USB card works well with this MB and in the VM. I did have one question though, do you see the same devices listed in the "safely remove device" section @Hoopster? How can I remove these from the eject list? Have I messed up the VM setup?

ejectDevices.png.96a4da8ce9d2c4113d4e978476ff1211.png

Edited by JimmyGerms
Link to comment
2 hours ago, Hoopster said:

This is a fairly standard list based on the way VMs work.  Here is the list I see with a few additions:

Whew, thought I may have messed up my template! Last question before I derail this thread any further. What are your temps like with the 2288G running a VM? I'm sitting at ~46C when my 6 core VM is idle. I feel like this is a bit too high for something like this. I'm so paranoid with temps and if I got good thermal paste coverage every time I build a PC.

 

Thanks for all the help @Hoopster! I've learned so much from this thread and building this machine!

Link to comment

Not directed to me, but I can share my experience with this CPU. It runs hot. Airflow and HSF will play a big role in keeping temps down, but this processor just has a high TDP. Even an idle Windows 10 VM seems to raise my overall CPU utilization by 5-15%, which subsequently increases temps. 
 

Using the best HSF that your case can support would be my recommendation. I was limited by case clearance so the HSF I use is a little small for my liking, but still does an admiral job. 
 

Personally I wouldn’t be concerned with the temp you mentioned just based on my own averages. I think an item that would be more meaningful is what are your temps at full load? 
 

 

Link to comment
1 hour ago, JimmyGerms said:

Whew, thought I may have messed up my template! Last question before I derail this thread any further. What are your temps like with the 2288G running a VM? I'm sitting at ~46C when my 6 core VM is idle. I feel like this is a bit too high for something like this. I'm so paranoid with temps and if I got good thermal paste coverage every time I build a PC.

 

Thanks for all the help @Hoopster! I've learned so much from this thread and building this machine!

 

Mine idles at 30C, though have a Noctua NH-D15S on it (140mm fan).

 

I also am paranoid about the CPU temps...so you are not alone.

Link to comment
1 hour ago, JimmyGerms said:

What are your temps like with the 2288G running a VM? I'm sitting at ~46C when my 6 core VM is idle

I am at 36C when both unRAID and the VM are idling; however, it does not take much to bump it up 20C and under high load, as others have mentioned, it can get quite hot.  I have seen temps up in the low 80s when all cores/threads are getting a good workout.

Link to comment

Just wanted to give a quick update on my experience with VMs. I got everything up and running and temps near the experiences here. Turns out the air printer tools I was using on the VM were poorly optimized and pinging the CPU threads I had isolated to 50% most of the time. Found another method to get our older printers up and running on the network for mobile devices. Thanks for all the help and data points!

 

Last question before I stop tinkering with settings. Does anyone else have trouble getting the BIOS time to stick? I thought I would update it through IPMI since daylight savings time but I'm finding after a reboot, shutdown or browser login the time is just wrong. I'm currently using 216.239.35.0 (time1.google.com) and 216.239.35.4 (time2.google.com) for the NTP servers but it seems to only grab the correct time when I hit "save". I don't think it's a huge issue but just curious if it's able to be solved?

 

Thanks again!

Link to comment
4 hours ago, JimmyGerms said:

Does anyone else have trouble getting the BIOS time to stick?

In Linux, BIOS time resets to UTC on reboot, etc. no matter what time you set it to.  Even if you set it to the correct time via IMPI, its going to go back to UTC on a reboot.

 

You can set your server time for unRAID to local time with the Dynamix Date Time plugin so times in the syslog, etc. are correct but BIOS time in Linux just wants to be at UTC.

Link to comment
On 3/19/2021 at 7:56 PM, DoItMyselfToo said:

i've been running the 2.21a bios and 1.80 bmc since i got my E3C246D4U.  seeing there is a 2.34 bios.  anyone know what the changes are from 2.21a?  anything critical?  microcodes?

Heard back from WIlliam at Asrock, who said that there is no reason to change to the 2.34 bios if the 2.21a is running fine.  Here is an exact quote:

 

”There is some minor modification in the BIOS to give little more support for a certain PCIe video card and there isn’t an additional feature added that would suit your need. If your system is current running fine with the current BIOS, you don’t need to use that beta BIOS. Thank you!”

Edited by DoItMyselfToo
quoted source.
Link to comment
On 3/9/2021 at 7:59 AM, parisv said:

will the nvidia 3000 and AMD 6000 series work ok with our motherboards (E3C246D4U)?

 

might buy one to crypto mine since the server is on 24/7

 

 

our boards don't fit full size gpus there's a heatsink in the way of the pcie 3 x 16 slot.

 

can use a rise cable and a vertical mount adapter if you have the right case. 

Link to comment
12 minutes ago, parisv said:

our boards don't fit full size gpus there's a heatsink in the way of the pcie 3 x 16 slot.

 

can use a rise cable and a vertical mount adapter if you have the right case. 

What cards will work then? Just the short ones meant for mini itx cases?

Link to comment

I can't seem to boot anymore. Bios picked up the usb ok if I select boot it can see the usb. I've tried copying the files from one usb to another then running the make boot batch file but still same issue. Error code on motherboard says A2 which means it can't find boot device. 

 

It's odd it just gets to the screen where is says press f2 for bios f11 boot menu and stays there. If I manually select boot device nothing happens. If I put the usb in another machine it loads the boot menu fine. I've also tried a windows usb iso too and same thing it just won't boot from usb I've tried different usb sockets too.

 

edit- working now it’s because I had a monitor plugged in. Doesn’t seem to boot with it in as I have the onboard graphics force enabled for Plex transcoding. I then got a pci error with a graphics card plugged in so I had to disable the boot inventory. Now passed through a 3070 to a window vm fine.

Edited by parisv
Link to comment
Posted (edited)
6 hours ago, parisv said:

Are you able to see your cpu temp and fan speed on the unraid dashboard?

 

No problem with CPU temp in dashboard:

 

image.png.0e08edcc01795d11a0f875d2bcf5111d.png

 

For fans, I use the IPMI Tools plugin and display the 2 HDD cage fans and CPU fan speeds in the GUI footer (note the incorrect MB temp of 87C):

image.png.115ce903f0d0d30bd07d977fa09f4f97.png

 

The CPU temp difference is due to different polling intervals.

 

Edited by Hoopster
Link to comment

My board is currently reporting as hot

 

from ipmi

201702993_Screenshot2021-03-28at12_29_31.thumb.png.9bdeb4aa017eda5045b0931698c4f947.png

 

ipmi plugin

1165497486_Screenshot2021-03-28at12_29_16.thumb.png.53ce9328d4a9f5360d0a7d7df206022a.png

 

Also I'm using a Fractal design meshify 2 which has a smart hub where I connect all the fans to including the 4 pin cpu fan then it connects to cpu_fan1. Problem is it's reporting as 0rpm and the red light next to cpu_fan1 on the motherboard is flashing. 

 

The fans which are plugged into the hub are spinning.

 

 

Link to comment
On 3/27/2021 at 3:36 PM, Hoopster said:

 

No problem with CPU temp in dashboard:

 

For fans, I use the IPMI Tools plugin and display the 2 HDD cage fans and CPU fan speeds in the GUI footer (note the incorrect MB temp of 87C):

 

The CPU temp difference is due to different polling intervals.

 

 

@Hoopster would you mind sharing a screenshot of your IPMI fan setup? I'm interested in your fan curves. I had to map mine out in a spread sheet to wrap my head around the settings haha.

Link to comment
8 hours ago, JimmyGerms said:

would you mind sharing a screenshot of your IPMI fan setup?

Frnt_Fan1 and Frnt_Fan 2 are my two HDD cage fans and are setup identically.  On Hard drives to poll, I just have the HDDs selected as they are ones in the cages and I have deselected the SSD/NVMe drives.

 

image.thumb.png.1f8ae3b81caf3147272202a7ec455a35.png

 

The way my case is designed, the air from the HDD cage fans flows through the HDD cages and then is redirected back over the MB.  I have set the minimum HDD fan speed to 50% to keep air flowing back to the MB and it helps with CPU and PCH temps (nothing can fix the MB temp reading) to have air flow at at least 50% even when all HDDs are spun down.

 

CPU Fan settings:

image.png.39b8a131692f20bb256052025529ac3c.png

  • Thanks 1
Link to comment
  • 4 weeks later...
On 4/26/2021 at 6:51 PM, Rick Sanchez said:

Has anyone thought about RocketLake?

You would need a Z590 (consumer) chipset based motherboard and a compatible Core i5, i7, i9 CPU

 

The latest server/Xeon chipset is W480 which, currently only supports Comet Lake 10th generation Xeon W-12xx CPUs. However, it is expected to support the new Rocket Lake W-13xx CPUs when they are released.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.