My SuperMicro Build

[uldise]

1 minute ago, demonmaestro said:

How does that help?

How did you do that?

i just watched at one card internal temp - one of my card is Dell H710P - true RAID card, and i can access it's internal chipset temp.

am i mounted Noctua NF-F12 PWM outside a case beside the cards with some tighteners, and provide fan cable back to case inside to connect them to the MB. i have no 2x80 fans on the back..   

[demonmaestro]

Well I got an CORSAIR Commander PRO, CL-9011110-WW. Basically an external fan controller. I am going to set some custom fan curves with temps and see if that helps. I just think the MoBo fan curve is just not there. I should be able to spin up the 120mm fans up to 1000 RPMs without making more noise and increase airflow.

Unless you think I am missing something?

[uldise]

2 minutes ago, demonmaestro said:

Well I got an CORSAIR Commander PRO, CL-9011110-WW. Basically an external fan controller. I am going to set some custom fan curves with temps and see if that helps. I just think the MoBo fan curve is just not there. I should be able to spin up the 120mm fans up to 1000 RPMs without making more noise and increase airflow.

Unless you think I am missing something?

if you increase fan wall RPMs, i think it does not help CPUs.. CPU temps should always stay at low, even with very high load... BTW, what CPUs are you using? mine's are X5680..

[demonmaestro]

Supermicro X8DTL-iF Motherboard

Dual Intel Xeon L5640 2.26Ghz Six Core Processors

96GB RAM (6x 16GB 8500R)

 

Here is the heatsinks for the cpus that I used. https://www.newegg.com/Product/Product.aspx?Item=N82E16816101300

 

The picture is of what all the fans at 100% look like with no load. (I did not have the fan on the raid cards at that given time).

[uldise]

i think you need better cooling for your CPUs - maybe, if they will be cool, then other fan's will not jump so high in RPM... but that is a guess only..

[demonmaestro]

It was the recommended cooler from supermicro for that mobo/cpu setup.

 

Hopefully @garycase will pop-in and help out.

[uldise]

look at this thread: 

may be that's is your case?

 

[demonmaestro]

So I did have that issue at first. However I had adjusted the lower limit of the ipmi to match the fans so no more surging like is mentioned in that post.

[garycase]

2 hours ago, demonmaestro said:

It was the recommended cooler from supermicro for that mobo/cpu setup.

 

Hopefully @garycase will pop-in and help out.

Can't resist "popping in" after getting a notification that showed you had requested a look

However, I don't see any obvious flaws in your setup.   Clearly the dual Xeon board is going to be FAR more power-hungry than your Avoton was.

 

What is your current motherboard/CPU configuration?   Did you move everything from the Dell to a rack-mounted case -- or have you built a new setup for the 4U case.    And is the 2U case just holding additional disks ... or is it a separate system (perhaps with your Avoton board)?

 

You certainly APPEAR to have very ample airflow for the system ... not sure why you're seeing the increased temps.   Perhaps the specifics on exactly what you've got mounted in the case may provide some clues.    Did you switch to higher-rpm disk drives?   Are you sure you used good thermal compound when mounting the CPUs?  etc.

[demonmaestro]

1 hour ago, garycase said:

Can't resist "popping in" after getting a notification that showed you had requested a look

However, I don't see any obvious flaws in your setup.   Clearly the dual Xeon board is going to be FAR more power-hungry than your Avoton was.

 

What is your current motherboard/CPU configuration?   Did you move everything from the Dell to a rack-mounted case -- or have you built a new setup for the 4U case.    And is the 2U case just holding additional disks ... or is it a separate system (perhaps with your Avoton board)?

 

You certainly APPEAR to have very ample airflow for the system ... not sure why you're seeing the increased temps.   Perhaps the specifics on exactly what you've got mounted in the case may provide some clues.    Did you switch to higher-rpm disk drives?   Are you sure you used good thermal compound when mounting the CPUs?  etc.

 

The smaller ATOM setup that I have is sitting in the 2u case running Esxi v6.5

With 2tb wd black, 4tb wd black, 120gb Samsung ssd, 960gb WD blue ssd 

 

the 4U case

14 hours ago, demonmaestro said:

 

Supermicro X8DTL-iF Motherboard

Dual Intel Xeon L5640 2.26Ghz Six Core Processors

96GB RAM (6x 16GB 8500R)

 

with. Using 2 - WD re 4tb for parity and 13 - WD Reds 4tb for storage and a 960 pny ssd for VMs.

I used Arctic Silver 5 thermal compound for the CPUs. Same stuff I have used on countless of systems and no issues.

Side note. With the HDDs spun up and the system idle it runs about 198w - 220w... (Not too bad..) the cpu only has a tdp of 60w

Edited June 2, 2017 by demonmaestro

[garycase]

What specifically are you concerned about?    The voltages look okay; and the temps don't seem overly high (unless I'm missing something).    Are the fans too noisy at the rpm's they're spinning at?

 

[demonmaestro]

23 hours ago, demonmaestro said:

4U case - at idle. 75-76F ambient temperature 

 

Sorry I know this has spanned a couple of pages... 

That is the temps at idle.... and the HDDs spun up.

 

Fan 1-2 = CPU

Fan 3 = Front of case 3 - 120mm noctua fans

Fan 4 = Middle of case 3 - 120mm noctua fans

Fan 5-6 = Back 2 - 80mm noctua fans.

 

There is a picture of the case and setup on page one.

Edited June 2, 2017 by demonmaestro

[garycase]

Interesting -- the temps are much higher; but also the fans are spinning at MUCH lower RPMs than in your "idle" picture about 9 posts ago.

 

Are the fan speeds now being controlled by the Corsair Commander?    If so, perhaps you need to adjust where you have the thermistors so the fans ramp up more when the system is under load.     Or perhaps raise the minimum fan speeds.    I suspect if the fans were running at higher speeds your temp problem would simply go away.   And with these Noctua fans, they'd probably still be pretty quiet.

 

 

 

[demonmaestro]

1 minute ago, garycase said:

Interesting -- the temps are much higher; but also the fans are spinning at MUCH lower RPMs than in your "idle" picture about 9 posts ago.

 

Are the fan speeds now being controlled by the Corsair Commander?    If so, perhaps you need to adjust where you have the thermistors so the fans ramp up more when the system is under load.     Or perhaps raise the minimum fan speeds.    I suspect if the fans were running at higher speeds your temp problem would simply go away.   And with these Noctua fans, they'd probably still be pretty quiet.

 

 

 

So the picture a few post back was from doing a parity check. For whatever reason when I had started the Parity check it spun up all the fans to max even though the cpu loads where at 2-3%.

 

I haven't gotten in the Corsair Commander yet. It should be in by Wednesday or Thursday next week.

I don't understand on why the mobo didn't raise the fan speeds on its own considering the fans are pwm

[garycase]

5 hours ago, demonmaestro said:

I don't understand on why the mobo didn't raise the fan speeds on its own considering the fans are pwm

 

Me either.   Does the BIOS allow you to set a range for the RPMs?   If so, perhaps that set way too low.   If not, then hopefully using the Corsair controller will let you get the thresholds set so the fans kick in when they need to.    It seems like that's the only real issue -- if the fans spin up when the temps start climbing I think they'll easily be controlled.

 

[demonmaestro]

We'll see once the controller gets in. I really don't want to buy a whole new mobo/cpu/ram..

But if I do I will upgrade to the 2011-3 socket. been eyeing a 10core xeon with dual cpu mobo. Just don't want to spend the $5k for a new bad to the bone setup.

I mean heck what I got is pretty nice.

 

Just was wishing I could make this all work without going that route.

the bios doesn't have a custom fan curve area.

Edited June 3, 2017 by demonmaestro

[uldise]

1 hour ago, garycase said:

It seems like that's the only real issue

@garycase i'm worried about CPU temps. look at the picture, with system idle CPU are Middle - that means cpu temp are max 10 degrees below overheat temp, according to intel site,

69.4°C is overheat temp for this CPU. again, i would suggest of changing CPU heat-sinks with some Noctuas. If Supermicro recommends those heat-sinks, then you need a SM case too, they have additional moding for additional airflow. i have seen SM case with passive CPU heat-sinks and some tunnels to case back to 2x80 fans..    

[demonmaestro]

41 minutes ago, uldise said:

@garycase i'm worried about CPU temps. look at the picture, with system idle CPU are Middle - that means cpu temp are max 10 degrees below overheat temp, according to intel site,

69.4°C is overheat temp for this CPU. again, i would suggest of changing CPU heat-sinks with some Noctuas. If Supermicro recommends those heat-sinks, then you need a SM case too, they have additional moding for additional airflow. i have seen SM case with passive CPU heat-sinks and some tunnels to case back to 2x80 fans..    

 

Originally the board had the passive cooling on it. I had put the fan heatsink coolers on it. But that is the model recommend from SM for the board.

 

Now the CPUs we're at medium when the system temp was at 57°c. 

So basically it was moving around hot air already onto the processor. 

 

But maybe @garycase has a different theory.

[garycase]

I'd see what the temps are with higher fan speeds under load => but I agree that if they're still in the 60 degree range I'd switch to a better CPU cooler.    But I suspect that simply getting more air movement is all you really need.

 

 

 

 

[demonmaestro]

Little Update.

I had looked into the bios and was presented with 4 options 

• Full Speed
• Performance
• Balanced
• Energy Saving

Come to find out it was on Energy Saving. HOWEVER I had tested it via the "fur mark" CPU test and with both CPUs at 100% load the fans were not spinning up. So I think the options is just preset voltage levels and not fan curves.

 

I had set the fans to Full Speed and ran the CPU test for 20 minutes and a parity check to load up the drives. The pictures below is the results of the temps.

Not too bad I think. Also the CPUs had went back to low little less than a minute after stopping the test. So it must of been in the lower temps.

 

So not sure as to what to do. Maybe the CORSAIR Commander PRO will help out somehow.

 

@garycase What do you think I should do moving forward?

[garycase]

The temps all look great with the full speed setting.    That's probably more aggressive than you really need.   I'd try the Performance and Balanced modes and see how they work.     You may not even need to bother with an external controller .

 

What's the noise level with full speed?      Ideally you'd like a setting that does a good job of cooling without blasting you out of the room from the fan noise

 

[demonmaestro]

32 minutes ago, garycase said:

The temps all look great with the full speed setting.    That's probably more aggressive than you really need.   I'd try the Performance and Balanced modes and see how they work.     You may not even need to bother with an external controller .

 

What's the noise level with full speed?      Ideally you'd like a setting that does a good job of cooling without blasting you out of the room from the fan noise

 

 

Well I had tested on all settings and "Balanced" is just a hair too much noise for my liking when idle.

Now full speed while it might not sound like a jet taking off  (like a poweredge 2950) it is WAY too much noise. lol 

 

The issue is it should ramp down the fans when at idle and speed up the fans accordingly with temp/load.

[uldise]

if you are on experiments now, i would try to disconnect front wall at all, then connect each of remaining fans to the motherboard directly without a splitters.. may be that's the case..

[demonmaestro]

19 minutes ago, uldise said:

if you are on experiments now, i would try to disconnect front wall at all, then connect each of remaining fans to the motherboard directly without a splitters.. may be that's the case..

But if that would make the system utilize a fan curve then wouldn't the Fan 1 and Fan 2 ramp up to 100% when the CPUs load up? 

Now the motherboard does not have labeled CPU header fans. It just has Fan headers #1 - #6.

 

I had read about a program "speedfan" but I did try to launch Core Temp to look at actual CPU temps in the windows VM and it had showed 0...

So I don't think speedfan would work unless there is another way?

 

Fan 1 = CPU 1

Fan 2 = CPU 2

Fan 3 = Front of case 3 - 120mm noctua fans

Fan 4 = Middle of case 3 - 120mm noctua fans

Fan 5 = Back 2 - 80mm noctua fans.

Fan 6 = 120mm noctua fan on top of raid cards.

[uldise]

just looked at your MB's manual, you have ok fan connections for CPU as Fan1 and Fan2 are for CPUs.

according Coretemp - you can't do it on VM i think, but on Host OS only.. you can boot some Ubuntu LiveCD, then run sensors-detect and it will show up. BTW, you can try sensors-detect un unRAID too.