Intel Socket 1151 Motherboards with IPMI AND Support for iGPU


Recommended Posts

I am interested in upgrading my server with a Xeon E-2278G CPU (they are finally beginning to appear in retail channels) which has an iGPU for video transcoding.  I am also interested in a server board with IPMI.  On many such boards, the BIOS does not support the iGPU when IPMI is active via the AST2X00 BMC with video output to a VGA port or JAVA/HTML5 console.

 

From this thread about the Supermicro X11SCA-F, we know that this board does support both IPMI and the use of the iGPU for transcoding.  There are things I don't like about that board, like the placement of the SATA ports stacked at the very edge of the board.  That would be a tight fit and cabling problem in my case.

 

I asked ASRock Rack if the E3C246D4U board supports both IPMI and the iGPU and they sent me the 2.10A BIOS which they claim implements that feature specifically.  I don't have that board yet, but, I am leaning towards purchasing one to pair with the E-2278G (when I can finally get my hands on one) and 64 GB RAM.  Not having the board yet, I cannot verify that it does what I need; however, I was very specific with ASRock about what I needed and they claim to have BIOS firmware to support it.

 

The E3C246D4U looks like a good option for my needs. 

 

For those interested in 10G NICs, the E3C246D4U2-2L2T motherboard has 2 10G NICs and likely also has BIOS fimware for IMPI + IGPU like its Gigabit NIC sibling.  That board seems impossible to find at the moment.

Edited by Hoopster
  • Like 1
Link to comment
On 1/31/2020 at 5:07 PM, Hoopster said:

I asked ASRock Rack if the E3C246D4U board supports both IPMI and the iGPU and they sent me the 2.10A BIOS which they claim implements that feature specifically.  I don't have that board yet, but, I am leaning towards purchasing one to pair with the E-2278G (when I can finally get my hands on one) and 64 GB RAM.  Not having the board yet, I cannot verify that it does what I need; however, I was very specific with ASRock about what I needed and they claim to have BIOS firmware to support it.

 

The E3C246D4U looks like a good option for my needs. 

Based upon my contact with the seller of this item: https://www.ebay.com/itm/ASrock-E3C246D4U-Server-Xeon-E-MB-w-2288G-CPU-64GB-ECC-RAM-256GB-NVMe-SSD-WOW/193237315094?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m1438.l2649

 

This is my conversation with him:

Screen Shot 2020-02-05 at 11.02.03 AM.png

Link to comment
  • 2 weeks later...

I just purchased the following combo:

  • ASRock E3C246D4U motherboard (best current price - $280 @ Newegg)
  • Intel Xeon E-2288G CPU (best current price $600 but the E-2278G is in stock @ Provantage for $483)
  • 64GB (2x32GB) ECC UDIMM DDR4 RAM (very hard to find and prices have shot up recently, best in stock price ($496 - 2x $248)
  • Samsung 256GB NVMe SSD (best price $70 @ Newegg)
  • Total price at retail - $1329 with an E-2278G, $1446 with the E-2288G

For $1099 it was a steal of a deal (see link in previous post, he has one combo left as of this post).

 

It should already have the BIOS for iGPU support simultaneous with the HTML5 IPMI.  Should be in late next week.  I'll let you know how it goes.

Edited by Hoopster
  • Like 1
  • Thanks 1
Link to comment
25 minutes ago, Hoopster said:

I just purchased the following combo:

  • ASRock E3C246D4U motherboard (best current price - $280 @ Newegg)
  • Intel Xeon E-2288G CPU (best current price N/A but the E-2278G is in stock @ Provantage for $500)
  • 64GB (2x32GB) ECC UDIMM DDR4 RAM (very hard to find and prices have shot up recently, best in stock price ($496 - 2x $248)
  • Samsung 256GB NVMe SSD (best price $70 @ Newegg)
  • Total price at retail - $1346 with an E-2278G

For $1099 it was a steal of a deal (see link in previous post, he has one combo left as of this post).

 

It should already have the BIOS for iGPU support simultaneous with the HTML5 IPMI.  Should be in late next week.  I'll let you know how it goes.

Very cool. Please let us know if you’re able to get QuickSync and Plex working with that Beta BIOS and 6.8.2 (I know it’s on 4.19.x but some people have reported 9th Gen QuickSync working).

Link to comment
13 minutes ago, ramblinreck47 said:

Very cool. Please let us know if you’re able to get QuickSync and Plex working with that Beta BIOS and 6.8.2 (I know it’s on 4.19.x but some people have reported 9th Gen QuickSync working).

Yeah, that's my concern.  I may have to roll back to the last 6.8.0 RC with the Linux 5.x kernel in order to get it to work.  I am eagerly awaiting the 6.9.0 RC.

Link to comment
5 minutes ago, kaiguy said:

How's your combo working out??

I'll know next week.  New power supply arrives on Monday and then I will be ready to do some parts swapping.

 

Yeah, I am really glad I bought that combo.  It ended up saving me over $300.  The Samsung 32GB ECC RAM modules are the only 32GB modules certified to work on this and other boards supporting the E-22xx processors and they are now incredibly hard to find and very expensive when you find them.

Link to comment

I put the new system together over the weekend to do some breadboard testing outside my case.

 

I bought a new power supply (Corsair SF600 Platinum) and CPU cooler (Be Quiet Shadow Rock TF2) and a two port USB 2.0 card that connects to the motherboard and put it together with the CPU, MB, RAM and NVMe SSD purchased in the combo.  I got the USB 2.0 card because unRAID boot issues seem to be reduced when booting from a USB 2.0 port as opposed to USB 3.x.

 

So far, so good. It all works.

 

I installed a new BIOS (the one ASRock sent me to enable the iGPU), made a few BIOS modifications and booted into vanilla unRAID with no problems whatsoever.

 

My big concern is that the i915 (Intel iGPU drivers) are currently a mess in the 5.x versions of the Linux kernel and a 5.x version may be required to use the iGPU for hardware transcoding with the Xeon E-2288G CPU.

 

UPDATE: Tom says 6.9.0 RC will include Linux kernel 5.5.6 or higher and general release is likely to contain 5.6 kernel.  Hopefully, i915 issues were resolved in the 5.5.x kernels.

 

UPDATE 2:  Apparently, Kernel 5.5 has all the same iGPU bugs as prior kernels.

 

Tonight, I will do the swap of this system into my case and connect the HDDs, SSDs, etc. from my current array and boot with my current configuration.  After that, I am sure there will be some tweaking of settings and I can do some iGPU testing as well as get IPMI fine tuned.

 

The adventure has begun.  More to come.

Edited by Hoopster
  • Thanks 2
Link to comment

I put my system in the case tonight swapping out the old components, but, now the BMC will not initialize.  Really strange.  Yesterday I had IPMI working perfectly and accessing the server via the HTML5 management interface (it's a beautiful thing).  I have contacted ASRock to see if they have some pointers.

 

On the plus side, hardware transcoding seems to be working with the Linux 4.19.98 kernel i915 drivers (hopefully this is not because the BMC is not initialized).

 

I forced a movie to transcode on my iPhone and it reports (hw) decode and encode with very little CPU activity even though audio transcode is done by the CPU.

 

image.png.1a1aa300bab1bb29db4ec276d0297ffa.png

 

Plex.png

Edited by Hoopster
  • Thanks 2
Link to comment

With the help of ASRock server support, I got my BMC issues resolved.  When I put the board in the case, I failed to notice that the UID button had not slipped though the hole for it in the IO plate as it was slightly out of alignment. It was stuck in a depressed position and, apparently, if the UID button is held down while the system is booted, that erases the BMC chip.

 

A side effect of this is the BMC controls PWM fans.  None of my PWM fans would spin at all. 

 

ASRock sent me a tool for flashing the BMC and once done, that resolved my BMC and PWM fan issues. 

 

I have dealt with this support contact at ASRock before (William Lee) and found him to be incredibly responsive and helpful. 

 

There is only one issue with these board and it is minor.  With the iGPU enabled through the latest BIOS ASRock has for this board, the monitor (and also the KVM in IPMI) goes blank at the end of the unRAID boot process when the go file is executed and 'modprobe i915 is run to load the iGPU drivers.

 

This is not a huge problem because all other IMPI functions are still available and if I need a console screen, I can run a terminal or PuTTY session rather than the KVM.  I believe Supermicro boards have a similar issue.

  • Thanks 1
Link to comment

Here's the latest update on my system configuration and performance.

 

When I flashed the BMC chip again, all PWM fans starting working again but I noticed the IPMI console would not report the fan speed and showed their fan sensors (Front Fan 1 and 2) as disabled.  I entered the BIOS and went into the H/W Monitoring section and it polled all the sensors again and activated the missing sensors.  The IPMI console was then happy and started reporting the fan speeds for these fans which cool my hard drive hot-swap bays.

 

I have now installed the IPMI Tools plugin in unRAID and have it configured to control these fans based on HDD temperatures.  The CPU fan is, of course, configured to react to CPU temperatures as well as some case fans.

 

So far, I am seeing decent CPU temperatures at idle on the E-2288G.  It fluctuates between 33-35C with the CPU fan at only 400 RPM.  I have IPMI Tools configured to show the CPU and two Front fans in the GUI footer and the System Temp plugin shows CPU and MB temps.

 

Here's a look at IPMI Tools footer configuration as well as all the temps in the GUI footer. The discrepancy in the reported fan speeds in the IPMI Tools config and the footer is due to polling time differences.

 

image.png.4c7c8df08529e1e71d680b83271d7333.png

 

When Plex is doing some 1080p transcoding, the CPU may get into the 40s as the on-die GPU is doing its thing.  I have yet to run a parity check with the new system or stress it with multiple loads, but, overall, it looks like temps will be well within decent temperature ranges. 

 

My next test is to plug the system into my Kill-A-Watt and monitor power draw at idle and under load.

Edited by Hoopster
  • Like 1
  • Thanks 1
Link to comment

And the results of one more test, power utilization with the system connected to a Kill A Watt meter:

  • During system boot - 60 W - 95W
  • System completed boot up and sitting at the unRAID GUI with all (5) HDDs spun up - 66W
  • System at idle with all HDDs spun down - 40W
  • During Parity Check (90 minutes in) - 80W - 90W with a max CPU temp of 42C and HDD temps between 34C and 39C
  • While transcoding three movies simultaneously - 75W - 80W with a max CPU temp of 38C

I am pleasantly surprised that my CPU Cooler is really doing a great job.  It's a Be Quiet! Shadow Rock TF2 top flow (C Style) cooler rated at 160W TDP.

 

 

Edited by Hoopster
  • Like 1
  • Thanks 1
Link to comment

Here is a test that puts a load on the CPU/GPU; Handbrake transcoding.  I have a preset that uses QSV and I am using that to transcode a file from .mkv to .m4v (mp4).  Without QSV, the CPU utilization would be in the high 80s to low 90s.

 

image.png.808a624adb4bdf86b31d29512f94a6ab.png

Edited by Hoopster
  • Thanks 1
Link to comment

Although the combo I bought for $1099 is no longer available (sold out and two of the three buyers are unRAID users), the same seller now has a similar combo available for $995.

 

It includes 16GB ECC RAM modules rather than the 32GB RAM modules in the combo I got.

  • ASRock Rack E3C246D4U MB
  • Intel Xeon E-2288G CPU
  • 32GB (2x16GB) Hynix ECC RAM - for $170 more, the RAM can be increased to 64GB
  • 128 GB NVMe SSD

I know the one combo that has been purchased was purchased by a member of these forums for his new unRAID build.

 

https://www.ebay.com/itm/ASrock-E3C246D4U-Server-Xeon-E-MB-w-2288G-CPU-32GB-ECC-RAM-128GB-NVMe-SSD-WOW/193362224931?hash=item2d05496723:g:OtgAAOSw7q5dv2bu

Edited by Hoopster
Link to comment
On 3/1/2020 at 4:01 PM, kaiguy said:

I purchased one of the new combos and upped the RAM to 64GB. Hoping it ships early next week. Thanks, Hoopster, for posting about your experience and guiding me through this process! 

Hi @kaiguy.  How did you go about adding the extra RAM to the purchase?  Did you just do the 'Make Offer' and added the extra in for the RAM?  I messaged the seller about this but I have not heard back.  I am really looking hard at this combo.  It is almost the same as what I am pricing out for my build, but it comes with the better CPU and 128GB NVMe.

Link to comment

@Burizado, I purchased as-is then messaged the seller letting him know I was waiting to make payment until he let me know what to do about the extra RAM. Later that evening he messaged me asking to complete the auction payment and he would send a separate PP invoice for remaining amount (which he did about a day later). I think he's just busy--there were times prior to my purchase when I'd ask a question via ebay and he would be very responsive, then answers to follow up Qs would be fairly delayed.

 

My combo should be arriving tomorrow!

  • Thanks 1
Link to comment
24 minutes ago, kaiguy said:

I think he's just busy--there were times prior to my purchase when I'd ask a question via ebay and he would be very responsive, then answers to follow up Qs would be fairly delayed

@Burizado This was my experience as well. He usually responded within 24 hours but it all depended on when I asked the question.  Most of the responses I got came late in the day or evening.  I asked a couple of questions and made an offer (rejected because of increase in RAM prices).  He responded once on a Saturday as well.

 

Once, he never answered, but, by then I had purchased the combo anyway and the question was moot.  

 

It sounds like the safest thing to do is what kaiguy suggested; purchase as is to make sure you get one and include a message that you want the RAM increased to 64GB and ask him to invoice you for the $170.

  • Thanks 1
Link to comment
On 3/1/2020 at 9:44 AM, Hoopster said:

Sold Out now!  @Burizado I assume you bought one of the remaining packages?  Let us know if your experience and test results vary significantly from mine (other than avoiding the mistake I made which resulted in my BMC chip being erased 😀).  It's a great unRAID platform, I hope you enjoy it.

  • Haha 1
Link to comment

Decisions decisions. Picked up 2 cpu coolers: the same Be Quiet that Hoopster used, and the Noctua NH-U9S (with the intention of returning one of them). Noctua support seemed to think that the NH-U9S may not be enough for this processor.

Quote

 

Q: Do one of your cpu coolers work well for an Intel Xeon E-2288G? One limitation, however, is I need the cooler to fit in a 4U case.

 

A: Well, "well" is not what I would name that, but "okay" would be the appropriate word. ;-)

The NH-U9S would fit, but it's not enough to handle the ~230W this cpu demands under full load. Only about 200W are possible, which is already pretty high but not enough for all core full boost potential.

 

I can't find any documentation that says the 2288G peaks at 230W... but now I'm leaning towards the Be Quiet since Hoopster has had good results.  Any thoughts?

 

Link to comment
40 minutes ago, kaiguy said:

Decisions decisions. Picked up 2 cpu coolers: the same Be Quiet that Hoopster used, and the Noctua NH-U9S (with the intention of returning one of them). Noctua support seemed to think that the NH-U9S may not be enough for this processor.

I can't find any documentation that says the 2288G peaks at 230W... but now I'm leaning towards the Be Quiet since Hoopster has had good results.  Any thoughts?

 

The Be Quiet would have even more issues if the CPU really peaks at 230W under full load.  The Be Quiet Shadow Rock TF2 is rated to handle up to 160W.

 

I do recall reading somewhere recently that the E-2288G draws closer to 130W at full load and this was an actual test of the processor.

 

As you can see from my Handbrake and power utilization tests a few posts earlier, the CPU hit 64C with a 62% load.  I did not have the system attached to the Kill A Watt at that time, so I don't know what the power draw was, but, clearly the cooler was handling the load fairly well given how active the CPU was.

 

The real question is how is the system going to be used and how often will it be under "heavy" loads?  I suppose I could stress test the CPU with a 100% load across all cores and see if the cooler can handle it, but, really, will there ever be a use case for this CPU that maxes it out like that?

 

I'll run a Handbrake test tonight WITHOUT QuickSync Video encoding (all CPU) and see what temps I get.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.