Intel Socket 1151 Motherboards with IPMI AND Support for iGPU


Recommended Posts

@Hoopster Oh nice. That’s super helpful. Thank you kindly!
 

Truth be told I’d probably rarely hit 100% if ever at all. I run a couple VMs and a lot of containers but given the 16 threads and QSV I think there’s plenty of headroom with this processor. 
 

Ok, now I’m leaning toward the Noctua... I think I’ll give it a shot tomorrow when I get my delivery! 

Link to comment
2 hours ago, Hoopster said:

Sold Out now!  @Burizado I assume you bought one of the remaining packages?  Let us know if your experience and test results vary significantly from mine (other than avoiding the mistake I made which resulted in my BMC chip being erased 😀).  It's a great unRAID platform, I hope you enjoy it.

Ha ha ha, yeah, I saw someone bought one today only leaving 1 left, so I decided to jump on it.  I wasn't sure if bumping this thread added more light on this deal, and was scared I would miss out (FOMO as the kids call it these days :D).  I bought all my other parts this afternoon as well.  Everything is supposed to arrive within the next week, with the MB/CPU set to be here Wednesday.

 

This is my first unRAID server, and even computer, build so I will test what I can and think is best, but I don't have any extensive tests in mind to perform.

  • Like 2
Link to comment

I ordered the Noctua NH-U12A for my setup, but I might have more space in the Fractal Design Define XL R2 than a 4U case.  I figured if it can handle the Intel Core i9-9900K with "best turbo/overclocking headroom" it would handle the E-2288G.  Since there was no real TDP stats on the E-2288G that I could find, I will see what my CPU temps are when I get everything setup.  If I need to look at a different cooler based on what my CPU temps are from testing I will.

Edited by Burizado
Link to comment

Here is a Handbrake transcode test of a 1080p movie from .mkv to .mp4 (don't have a HEVC 4K file handy).  I selected a preset that does not use QSV so this is all CPU.  I even told it to burn in subtitles for more CPU load.  This is 20 minutes into a 30 minute transcode operation:

 

image.png.9693e38e734dae2d0d69b3c706d52297.png

 

This is more than halfway through the transcode, so the CPU has been under load for a while.  The highest I saw the load go was 94%.  CPU temps remained in the mid-high 70s; that's a very respectable temp for the amount of sustained load on the CPU.

 

I do not have it connected to the Kill A Watt but my UPS reported a 72W load before I started.  During the transcode it rose to 210W. 

 

image.png.39897caeee6ae0fe24c5978121e66d87.png

 

That's as stressed as I imagine the CPU will ever get with my uses of the server.  Overall, I think this is very reasonable.

Edited by Hoopster
  • Thanks 3
Link to comment
14 hours ago, Burizado said:

If I need to look at a different cooler based on what my CPU temps are from testing I will.

You should be fine with that cooler. 

 

Because of the case in which my system is installed (Silverstone CS380) I had a CPU cooler height limitation of 146mm.  There are not many good tower coolers under that height (the Cryorig H7 Plus at 145mm is the only one with 120mm fans) so I ended up with a Be Quiet! Shadow Rock TF2 top flow cooler with a 135mm fan and a 112mm height.  Fortunately, it does live up to its name.  It is quiet even at full speed.

 

As you can see from my latest Handbrake test, that cooler did an admirable job in keeping CPU temps in the 70s even under full load.  I imagine the NH-U12A will do even better.

 

I made the mistake of buying the Cryorig before I settled on the MB choice and it does not fit well on the E3C246D4U because of the RAM placement.  

  • Thanks 1
Link to comment

Migration complete! The most challenging parts stemmed more from booting back into unRAID and making sure my device assignments were correct (the first time none of my SSDs showed up. Seems I had a bad breakout cable, so I had to swap it out. Luckily I had an extra one). For some reason unRAID also thought I had an unclean shutdown (I didn't), so I had to cancel the parity check. I'll probably kick it off again in a day or two once I've had time to play around with everything. So far all seems to be running well! The IPMI on this board is fantastic--so much better than my old SM mobo that required an outdated java client.

 

Idle temps look great so far with the Noctua NH-U9S paired with an IC graphite thermal pad instead of thermal paste. It was a bit of a gamble on my part, but everything I've read said it really only loses about 2C compared with high quality paste, with the benefit of it not having a limited lifespan like paste. So far so good.

 

@Hoopster did you have to do anything special to get the Dynamix temp plugin to detect the "array fan speed"? All I get is "Not used" and am not presented with any options. Also, oddly, it is constantly reporting 32C. The sensors it detected were coretemp and nct6775. Even when stress testing my CPU with a video encode, it stayed at 32C, whereas the IPMI support plugin showed it hovering between 70-74C. Looks like our cpu coolers are both performing well!

 

Col58xP.png

 

Also, are there vulnerability mitigations that I need to disable in the BIOS? Despite my having installed the plugin, I still seem to have some enabled.

 

Looking forward to playing around with everything some more in the coming days. Super excited about this upgrade.

Edited by kaiguy
Link to comment
8 hours ago, kaiguy said:

did you have to do anything special to get the Dynamix temp plugin to detect the "array fan speed"?

The Dynamix temp plugin has never shown me the fan speeds, it just shows 'not used'.  I get just CPU and MB temp from that.  In my prior (non-IPMI) system I used the Dynamix Autofan plugin to control and display the fan speeds.  However, with your new motherboard with IPMI, IPMI controls the PWM fans and you need to use the IPMI Tools plugin.  I use it to display CPU fan speed and the speed of the PWM fans on my hard drive hotswap cage so I can control them based on HDD temps.

 

In the screen snip below, the temps are coming from the Dynamix Temp plugin and the fan speeds from IPMI Tools.  Actually, IPMI Tools will let you designate up to four temps/speeds in the footer so you can get CPU and MB temps as well as two fans just from IPMI Tools.  I want the five I mentioned, so, I use a combination of the IPMI Tool and Dynamix Temps.

 

image.png.b600258fce4484806ce4631cf4ef8a65.png

 

If your CPU temp is really 31C under a 92% load across all cores, I need to get whatever CPU fan you are using 😁!

Link to comment
2 minutes ago, Hoopster said:

If your CPU temp is really 31C under a 92% load across all cores, I need to get whatever CPU fan you are using 😁!

It's not... whatever Dynamix temp is polling for CPU temp is not what IPMI is reporting. I did find another option (coretemp instead of cpu temp) which is closer, but still doesn't line up with what IPMI is reporting for CPU temp. Dunno.

Link to comment
8 hours ago, kaiguy said:

Also, oddly, it is constantly reporting 32C. The sensors it detected were coretemp and nct6775

Does the Dynamix Temp plugin show all these sensors in the drop down?

image.png.5b49009d24ee4c243df766dd466f9ee4.png

 

If any sensors you think you should be seeing are missing here or in the ASRock IPMI console, you may need to go into the BIOS H/W Monitoring screen and reread the sensors from there.  Then they should show up elsewhere as well.

 

Link to comment
9 hours ago, kaiguy said:

Also, are there vulnerability mitigations that I need to disable in the BIOS? Despite my having installed the plugin, I still seem to have some enabled.

I don't think the plugin currently enables EVERY mitigation that is out there.  It seems to focus on Meltdown, Spectre and Zombiland.  If you see this, that's everything I have disabled:

image.png.0f4d223f7d8fbd6a505f6fc09eb1ec23.png

 

It shows up like this in syslinux.cfg:

image.png.832b5e4a722539f1bfb2832e41b61548.png

Link to comment
12 minutes ago, Hoopster said:

I don't think the plugin currently enables EVERY mitigation that is out there.  It seems to focus on Meltdown, Spectre and Zombiland.  If you see this, that's everything I have disabled:

image.png.0f4d223f7d8fbd6a505f6fc09eb1ec23.png

 

It shows up like this in syslinux.cfg:

image.png.832b5e4a722539f1bfb2832e41b61548.png

Mitigations that are handled via microcode will still show as being mitigated.  Those shouldn't affect performance anyways

  • Thanks 1
Link to comment

@Burizado Thanks! Yep, I contacted ASRock after I bought the combo to request the beta BIOS. They replied fairly quickly with a link to download.

 

Also, while not directed to me, both VT-x and VT-d were enabled by default in the BIOS. The problem I made was configuring the BIOS first then flashing. Some of the settings didn't seem to stick (explicitly enablling the onboard graphics, for example). I'd recommend flashing first thing, then configuring. Flashing is super simple. Throw the binary onto a flash drive, plug it in, and select the option for Instant Flash during POST.

  • Like 1
Link to comment
3 minutes ago, kaiguy said:

@Burizado Thanks! Yep, I contacted ASRock after I bought the combo to request the beta BIOS. They replied fairly quickly with a link to download.

 

Also, while not directed to me, both VT-x and VT-d were enabled by default in the BIOS. The problem I made was configuring the BIOS first then flashing. Some of the settings didn't seem to stick (explicitly enablling the onboard graphics, for example). I'd recommend flashing first thing, then configuring. Flashing is super simple. Throw the binary onto a flash drive, plug it in, and select the option for Instant Flash during POST.

Ah ok.  Thanks for the quick reply update!

 

Yeah, my plan is:

  1. Setup hardware
  2. Flash BIOS
  3. Config BIOS
  4. Install unRAID
  5. Config unRAID

My hardware isn't due until early to mid next week, so I am getting the space ready near my modem/router in the basement.  Then will be feverishly awaiting the UPS delivery. :D

Link to comment

@Hoopster If at all possible, would you please identify which Samsung RAM modules came with the bundle you got off Ebay.  Since the bundle is currently not available on ebay, I'm also looking at other options.  I was wondering if the Samsung 32GB stick is the exact one mentioned in AsRock's qualified list, or if other versions work.  Server memory starts to make my head spin looking not only at speeds, but Rank x Org.

Link to comment
On 2/28/2020 at 11:27 PM, Hoopster said:

The BIOS that supports the iGPU in addition to BMC is only available directly from ASRock support.  It is BIOS L2.10A.  The latest BIOS available for download on their site is P2.10.

hello , I am using the E3C246D4U broad + i3-9100T cpu , so can't open the iGPU , could you please upload the BIOS L2.10A to me for convenience ? [email protected]

Link to comment

I guess our eBay friend has discovered that we unRAIDers like his bundles.  I know at least 4 of the 6 that purchased his last two bundle deals were members of this forum.  

 

He has listed another set of three of the last bundle he offered (E3C246D4U, E-2288G, 32GB (2x16GB) Hynix ECC RAM with an option to upgrade to 64GB, 128GB NVMe SSD.

 

https://www.ebay.com/itm/ASrock-E3C246D4U-Server-Xeon-E-MB-w-2288G-CPU-32GB-ECC-RAM-128GB-NVMe-SSD-WOW-/193377374775?_trksid=p2385738.m4383.l4275.c10

Edited by Hoopster
  • Like 1
Link to comment

Everything has been smooth sailing here! Digging this setup knowing I have the power when needed, but even with multiple Plex transcodes and QSV, the CPU sits pretty idle the majority of the time.

 

Only two items that have stumped me so far:

 

1. The mobo temp stays very consistent in the mid 20C-range. However yesterday I got a notification of critical temp, and saw that IPMI was reporting the mobo temp at 87C degrees. The server was completely idle, CPU temp was 31C, and the fans were operating as normal (which I have set to be operating at 75% of full speed at a minimum. Had to reboot the server to correct and haven't seen the same since. Stuck sensor, maybe?

 

2. Despite having IPMI plugged in via its dedicated port, I am not able to access the HTML5 IPMI GUI from the server itself (when at work I utilize Guacamole to RDP to a Win 10 VM or Firefox container). Not a showstopper since I can connect via openvpn to my router if needed, but it is odd. I thought that would only be an issue with a shared LAN/IPMI port.

Link to comment
The mobo temp stays very consistent in the mid 20C-range. However yesterday I got a notification of critical temp, and saw that IPMI was reporting the mobo temp at 87C degrees. The server was completely idle, CPU temp was 31C, and the fans were operating as normal (which I have set to be operating at 75% of full speed at a minimum. Had to reboot the server to correct and haven't seen the same since. Stuck sensor, maybe?

I saw something similar during a parity check; MB temp shot up to 84C and reported critical in IPMI console.  Normally my MB temp shows 31-32C.  I have only seen that once and after a reboot it went back down to to "normal" range and has never climbed that high again.  I don't know if that was related to a parity check but I have never seen that happen with any other activity (including HandBrake stress tests) on the server.    

Interestingly the AUXTIN sensors all report temps from 59C up to 83C continually.  They don't seem to change much.  I have no idea what those monitor.  When the MB temp reported 84C, AUXTIN3 reported the same temp.  I think there may be some "anomalies" in the sensors and their reporting.

Link to comment

I received all my parts, and first POST was good!  I looked around in the 2.10 BIOS for a bit then Instant Flashed the L2.10A.  Is there anything special in the L2.10A BIOS that I need to set to have IPMI and the iGPU working in unRAID?

 

I am waiting on some new network equipment, that should be here tomorrow, before I can get the server online.  I do have the unRAID USB setup, but not much I can do with that until the server is networked up.  Since my router doesn't support LAG, is there a point to connect both LAN ports?  I was planning to just connect 1 LAN and the IPMI to the router.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.