Intel Socket 1151 Motherboards with IPMI AND Support for iGPU


Recommended Posts

OK.  I got everything finally setup, and mostly working the way I want (basic media server setup with Plex).  I did get HW transcoding to work, once I figured out the correct option in Plex to check.  Initially the CPU was around 50-60% transcoding 1 movie.  Then when I enabled HW transcoding it dropped down to 5-6%.  I am doing RAM transcoding as well (at least I think I am, I have not seen any transcode files in /tmp yet).

 

One thing I have not gotten to work though is the IPMI interface.  I am assuming I am doing something totally wrong since I have never used this type of setup before.  I've been googling around some to try to find my answers, but I figured I'd post back in this thread since there are a few in here with similar setups ;).

 

Based on the IPMI doc from ASRock, it sounds like I can just use the IP of the IPMI port and port 4433 (for me I am using 'https://192.168.1.119:443').  I figured I would try just to straight connect before setting up any security, to try not to complicate things while learning something new ;).  It just goes to a page that says the site can't be reached.  Is there something I am missing?  I do have my router setting a static IP to the BMC, so I have the BIOS set to no for 'Manual Setting IPMI LAN'.  I do need to reboot my server so I will double check that the 'Out of band Access' is enabled.

 

Also, to note is I have the IPMI plugin loaded in unRAID, and I can see the fan information in the plugin.  So it must be getting some information from IPMI.

2020-03-16 20_18_09-E3C246D4U-IPMI.pdf - Adobe Reader.png

Link to comment

@Burizado You can disregard all that SSL stuff unless you intend to install an SSL certificate.

 

I am just pointing my browser to https://{ip address of IPMI NIC}  The :443 wont work unless you have an SSL certificate.

 

I do have the IPMI LAN set to manual in the BIOS with the same IP address I have statically assigned in the router. 

 

To initially connect with IPMI I may have used http:// with no 's' and then gone to advanced in the "non-secure site" warning in the browser and accepted the risk.

 

Once the IPMI page came up and I logged in admin/admin by default, I bookmarked the page and changed the admin password.  Here's what the bookmark shows:

 

image.png.f37ce3512fc6415a168a3993b03c2178.png

  • Thanks 1
Link to comment
42 minutes ago, Burizado said:

I am doing RAM transcoding as well (at least I think I am, I have not seen any transcode files in /tmp yet).

I am also transcoding in RAM.  Let me know if you think it is not working (you should have Trascode/Sessions folders under /tmp if it is working).

Edited by Hoopster
  • Thanks 1
Link to comment
On 3/16/2020 at 9:06 PM, Hoopster said:

I am also transcoding in RAM.  Let me know if you think it is not working (you should have Trascode/Sessions folders under /tmp if it is working).

I know it's not the subject of this thread, but I figured out my issue with RAM transcoding.  I am using the binhex-plex docker and the transcode directory variable in the default setup was set as a 'variable' and not a 'path' config type.  I deleted the one that was there and recreated as a 'path'.  Then I started seeing the files in /tmp/Transcode/Sessions.

 

If you notice the 'variable' doesn't have the 'Access Mode' option which might have been the reason.  Not sure, but it is working now.

2020-03-18 07_51_29-Atlas_UpdateContainer.png

2020-03-18 08_03_11-Atlas_UpdateContainer.png

Link to comment

@Burizado Glad you got it sorted! Personally I switched from /tmp to /dev/shm for transcodes only because with 64GB I can't imagine needing more than 32GB of transcode space. Still not sure what best practice is (it used to be /tmp), but its been working well for me.

 

Curious, how are your temps with your bigger Noctua tower cooler? My CPU temps seem to bounce all over the place. Idle in the lower 30s, but under any load it jumps around between 40-80C. Could be my smaller cooler, or the fact I used a thermal pad instead of paste. Or maybe because I have Smart Fan Control for most of the fans. But I'm interested to get some more datapoints on this processor. Thanks!

Link to comment
10 minutes ago, kaiguy said:

My CPU temps seem to bounce all over the place. Idle in the lower 30s, but under any load it jumps around between 40-80C

I'm seeing the same thing.  It rarely ever exceeds 80C even under sustained heavy load but it does tend to bounce around a lot under load. 

 

Since I moved the system into a cabinet (with ventilated door and sides, my idle temps creeped up a bit into the 35-38C range.  Out of the cabinet, they were 33-35C.

 

The only time I have seen high 70s and brief journeys into the low 80s is with sustained CPU-based HandBrake transcoding and, recently with my attempts to run BOINC on a sustained basis with some COVID-19 projects.  With iGPU transcoding it stays in the 50s maybe up to low 60s

 

What does concern me is that I had three hard lockups on Monday and into Tuesday morning with BOINC running a heavy load anywhere from 2 to 6+ hours before the system completely froze and I had to do a hard reset via IPMI.  As far as I can tell, CPU temps were not the problem; they bounced around between the 50s up to 80C. 

 

I wonder if PCH temps are too high (always seem to be in the 70s) or if I have some voltage issues.  I am going to try a 40mm fan on the PCH heatsink to see if it makes a difference.

Link to comment

@Hoopster I saw that you were experiencing lockups. I can't speak to the original container, but I manually installed the new linuxserver boinc and ran it all last night on 4 cores/8 threads. Luckily no lockups for me so far (knock on wood). I did do the pinning and the recommended settings, as well as setting some manual processor limitations within the app. Maybe try the new container?

 

This is me right now:

 

wQqU6FL.png

  • Like 1
Link to comment
1 minute ago, kaiguy said:

I can't speak to the original container, but I manually installed the new linuxserver boinc and ran it all last night on 4 cores/8 threads. Luckily no lockups for me so far

Would you mind sharing a screenshot of your manual container settings in advanced mode?  I gave it a brief go and the container downloaded OK, with, I think the proper settings but I could not get logged into GUI and it said it was running but nothing was happening.

 

I didn't have a lot of time to mess around with it so I have not tried again.  I am sure I messed something up.

 

Have you tried to set it up to use the iGPU? 

Link to comment

@Hoopster Sure thing. I've attached it since it's pretty big.

 

I did not setup the GUI link within the container setup yet, so I just manually go to my IP:port.

 

Edit: Yes, I did passthrough the GPU, but I'm not entirely sure Rosetta is using GPU at this point.

 

boinc advanced settings.png

Edited by kaiguy
Added GPU info
  • Thanks 1
Link to comment
4 hours ago, kaiguy said:

@Burizado Glad you got it sorted! Personally I switched from /tmp to /dev/shm for transcodes only because with 64GB I can't imagine needing more than 32GB of transcode space. Still not sure what best practice is (it used to be /tmp), but its been working well for me.

 

Curious, how are your temps with your bigger Noctua tower cooler? My CPU temps seem to bounce all over the place. Idle in the lower 30s, but under any load it jumps around between 40-80C. Could be my smaller cooler, or the fact I used a thermal pad instead of paste. Or maybe because I have Smart Fan Control for most of the fans. But I'm interested to get some more datapoints on this processor. Thanks!

I haven't really seen the CPU temp above mid-30C (34C I think the highest), but typically is in the 27-30C range.  I m not doing anything heavy yet, like HandBrake though.  The CPU fan sits at 700rpm, and only once I saw it drop to 500rpm.  I should try pushing it a bit with the BOINC software to see where it will take it.  So far I am still setting up things like VMs now.

  • Like 1
Link to comment
4 hours ago, kaiguy said:

@Hoopster Sure thing. I've attached it since it's pretty big.

 

I did not setup the GUI link within the container setup yet, so I just manually go to my IP:port.

 

Edit: Yes, I did passthrough the GPU, but I'm not entirely sure Rosetta is using GPU at this point.

Thanks, this got me going.  Yeah, it appears Rosetta is not using the iGPU.  I'll let it run a while but I suspect I will be locked up at some point in the next few hours.

 

image.png.b585a4feb0d2a3727bd9ecc0ff0754d8.png

Link to comment

 

@kaiguy I'm pretty sure it is the MB temp (PCH?) that has been locking up my server.  It has climbed from 32C to 78C in the 50 minutes I have had BOINC running.  I'll keep watching it, but I expect a lockup after a while.  Still no fan on the PCH.  IPMI is also now inaccessible.  Web page comes up but login attempts just spin for a few seconds and come back to the web page.

 

On the plus side, it appears that the rb_03_18_xxxxx tasks I am running now are using the iGPU as CPU load has really dropped (I am seeing just gray and yellow bars) and the wattage consumed by the CPU has also dropped significantly (as reported by UPS wattage load).  It was at 140W+ and is now holding steady at 78W.

 

By suspending any non rb_ tasks the Rosetta@home project gave me others and I now I just have rb_ task which are more likely to be the COVID-19 tasks, according to Squid.

 

UPDATE:  And...BOOM! After 80 minutes running...another lock up!

Edited by Hoopster
Link to comment
10 hours ago, kaiguy said:

@Burizado Glad you got it sorted! Personally I switched from /tmp to /dev/shm for transcodes only because with 64GB I can't imagine needing more than 32GB of transcode space. Still not sure what best practice is (it used to be /tmp), but its been working well for me.

Yeah, I saw the posts about using /dev/shm over /tmp but I didn't quite understand the difference.  Aren't they both using RAM?  Is it just that /dev/shm is limited on the amount of space it is allowed to use?  Since you mentioned 32GB out of your 64GB, I am assuming it uses half?

Link to comment

So a question with this MB and CPU setup, does anyone have both LAN taps plugged in with a router that does not support LAG?

 

My thinking is I can dedicate one port for Plex, and the other for everything else.  Would this be possible and is it worth the setup hassle, or would i not see much benefit out of it?  I didn't want my other network activities to start affecting my Plex streams.

 

With only 1 or 2 1080p streams on Plex at a time I can't image I am pegging the 1GB port, but thinking for future use.

Link to comment

@Burizado yeah, it’s 50% of RAM. 
 

Re: LAN ports, I have not, and everything I’ve read is it’s not worth the hassle. Too many complications for extra bandwidth that’s not needed 98% of the time. Not saying you shouldn’t try, but I’ve never felt the need to. My drives don’t exceed gigabit anyway—even my SSDs. 

  • Thanks 1
Link to comment

It looks like I have solved my stability issue under sustained load.  BOINC has been running now for 15+ hours and all systems are go.

 

As I posted previously, the MB temp had risen to 78C right before the crash.  Usually it is in the low 30s no matter what I am doing (other than heavy sustained loads).

 

I have two 120mm intake fans on the side of my case that blow air through the drive cages to the other side of the case.  After the air passes through it has nowhere to go but back to the motherboard.  Reading up on the case design, this was intentional as the air flows in under the motherboard to cool it.  There are two MB related temps in the IPMI console, MB  and Card Side Temp.  Apparently there are sensors on the bottom and top of the motherboard.  Card Side Temp is the top (the side where cards are inserted).  This temp has always been 39-40C.

 

I had set the two side panel intake fans via IPMI tools to react to HDD temps.  When no hard drives are active, the fans were spinning at a mere 200 RPM.  At 35C, they spun at 700 RPM and at 40C they kicked on full at 1400 RPM.  The problem was that if BOINC was the only activity on the server, the HDDs were all spun down and the fans were spinning at a mere 200 RPM; not enough to provide any cooling to the MB.

 

I now have the fans set to spin at full speed constantly (at least for as long as I am running BOINC).  HandBrake transcodes finished fast enough (unless I had several queued up), that the MB would never have heated up enough to crash the system even when all HDDs are spun down.  My transcodes go to SSD.

 

For the 15+ hours that BOINC has been running, the MB temp has stayed between 30-35C with the intake fans on full speed.  There is another intake fan on the bottom of the case that blows air up over the motherboard, but, by itself, that was not enough cooling.  I also have a 40mm fan on the PCH (C246 chip) heatsink but that has really not affected the temps much.  It is always around 71C.  At night it when it is cooler in the house, I saw it dropped to 69C.  Without the fan it was 71C.

  • Like 1
Link to comment

Thanks for sharing, @Hoopster. One thing I did with the ipmi plugin for my fans pulling air across the hard drives (and into the case)... I do have it linked to HDD temp, but I set the minimum percentage to a higher level (currently 70%, since my Noctua fans are pretty darn quiet). That way I'm always getting adequate airflow into the case, and only bump it up to above 70% when the HDDs need additional cooling. So far I've found this to be the best balance for my case and temps.

  • Like 1
Link to comment

Bought one of the 2288G Combos that @Hoopster posted. Looking forward to getting my first unRAID Build set up next week. Got the upgrade to 64GB RAM. I have a 450W power supply, hoping that is going to be enough for the build. Will post more info once I actually get it built, as of right now everything is in the process of being shipped.

 

Build:
Define R7 XL Case - ASRock Rack E3C246D4U Mobo - Xeon E-2288G CPU - 64GB Hynix ECC DDR4 RAM - Noctua NH-U12S CPU Cooler - Corsair SFX450W Platinum PSU - 2x10TB WD Shucks for Parity, 4x10TB WD Shucks for storage, 2x 1TB Samsung 860 EVO Cache Pool - LSI 9207-8I HBA.

Link to comment
35 minutes ago, Ewok said:

I have a 450W power supply, hoping that is going to be enough for the build

That should be plenty.  Until I did this upgrade, I had the same Corsair 450W SFX PSU in the case.  I upgraded to the 600W version, but, that was probably not necessary.  I only did it with the future in mind, not that I have specific plans, just wanted the flexibility to throw more at it.

 

Frankly, with the 2288G running under a heavy load, 5 8TB HDDs all spun up, 2 SATA SSDs, an NVMe SSD, an optical drive, 4 case fans running on full speed and the CPU fan at full speed as well, the most I have ever seen the build consume wattage wise is around 240-250 watts.

 

Unless you add a couple of power hungry GPUs that need some serious juice from the PSU, your 450W PSU is plenty even if you add a lot more hard drives.

Link to comment
1 hour ago, parisv said:

I've been following this thread, thanks for the info so far.

 

I'm going for the ASrock rack E3C246D4U and the Xeon 2244G. I want to use my old fractal design mini case I bought in 2011. Can you see any issues? 

 

https://www.fractal-design.com/wp-content/uploads/2019/07/Define-Mini-PS.pdf

I don't see any issues with putting the E3C246D4U in that case.  it says it takes mATX motherboards, has up to six fan slots and can take CPU coolers up to 160mm in height.  You should be able to use a nice cooler.  The E-2244G is rated at a low 71W TDP (compared to 95W with the top-of-the-line E-2288G) so you should be able to cool well any processor you choose to install on the board.

 

The only "limitation" in that case is the 6 HDDs/SSDs (8  I guess if you put 2 more in the 5.25" bays).  As long as you are OK with that, there should be no problem.

Link to comment

Welcome to the club  parisv!

 

One issue I have been looking into is trying to get audio from my Win 10 VM.  I noticed that the E3C246D4U does not have audio out jacks on the Rear I/O, and I don't see a mention of an audio chip on board in the specs.  Is that the case for others that have this board?  Has anyone got audio working in a VM?

 

For reference I am not trying to pass the iGPU through since I am using it for Plex transcoding, but I didn't think that would effect the audio for the VM.

Edited by Burizado
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.