Jump to content
Shamalamadindong

$90 Xeon E5-2670 2.6Ghz (8cores / 16threads)

1137 posts in this topic Last Reply

Recommended Posts

 

Did you update your port forwarding for SSH access in your router? (assuming your internal IP has changed with the new server)

 

Thank you Drider. I left internal IP the same... I VPN through my router and in essence, at work, I am on my home network. I can see my shares, webGUI, etc. Only thing its not doing is connecting via Putty. If I am on one of my computers at home, I can putty with no problem. Only fails when going in through the VPN.

 

SSH access would have not change because of a board change, it would if you changed the IP. Could your router be blocking SSH?

 

Yeah it makes no sense that it would be the motherboard. I can ssh from inside my home network.... I will try to muddle through it. Thanks you guys.

 

If anyone runs into this issue; I solved it in Putty changing from SSH to Telnet. This connects fine.

 

32Cjg1a.png

 

I am happy now.

 

:D

Share this post


Link to post

Dual 750 platinum PS awesome.

 

16x2.5 hotswap, hmm. Not many of us are using 2.5 drives for the array yet.  Are you going to fill it with 2.5 drives?

 

Glad to hear it's quiet.

 

The same seller also has the redundant 750 watt chassis with 8x3.5" HSBP (with and without redundant chassis fans).  But yeah, I'm going to load this new box with lots of 2.5" drives (won't be using unRAID), this'll be a dedicated ESXi box.

 

I purchased the same chassis and air duct.  I also bought the 8x3.5 tray separately for  $40 and swapped out the 2 2.5 drive trays.  I agree with the hairdryer description of the noise.  I've only booted into the bios so far but setting the acoustic levels hasn't improved things.  My 2600cp is on an ancient bios so I'm hoping that will make a difference.  If not I'll swap out fans and put a different powers supply in .

Share this post


Link to post

Dual 750 platinum PS awesome.

 

16x2.5 hotswap, hmm. Not many of us are using 2.5 drives for the array yet.  Are you going to fill it with 2.5 drives?

 

Glad to hear it's quiet.

 

The same seller also has the redundant 750 watt chassis with 8x3.5" HSBP (with and without redundant chassis fans).  But yeah, I'm going to load this new box with lots of 2.5" drives (won't be using unRAID), this'll be a dedicated ESXi box.

 

I purchased the same chassis and air duct.  I also bought the 8x3.5 tray separately for  $40 and swapped out the 2 2.5 drive trays.  I agree with the hairdryer description of the noise.  I've only booted into the bios so far but setting the acoustic levels hasn't improved things.  My 2600cp is on an ancient bios so I'm hoping that will make a difference.  If not I'll swap out fans and put a different powers supply in .

 

Running through the BIOS update (and the subsequent FRUSDRUPDATE) *will* reconfigure the BMC for your chassis configuration and allow the BMC to properly control the cooling domains and substantially slow the fans down.  This is mandatory for proper operation.

 

 

Share this post


Link to post

so have you guys with dual 2670 have you sorted out the cpu pinning yet? im getting unusual results when running multiple vm's

 

Share this post


Link to post

Damn it, I ordered the incorrect Noctua heatsink/fan (NH-D14) and didn't realize that I needed to also order the bracket alongside it (NM-I2011). Got that ordered and it should arrive today. Should have my rig built over the weekend. Can't wait.

Share this post


Link to post

Noctua certainly knows what they are doing when it comes to cooling!!

 

When I first installed the dual Xeons, I used the SuperMicro SNK-P0048AP4 2U Active DP CPU Heatsinks.  Under full load, they were seriously loud. Even under light loads, they had a high-pitched whine to them.  I can handle fan noise, but these were ridiculous!

 

I purchased a pair of Noctua NH-D9DX i4 3U from Amazon for $54/each. These things are enormous!  But, I when it comes to cooling, that is really what you want.

 

The instructions tell you to remove the fan before installing the heatsink, so I did. The heatsink itself fit just fine. However, I could tell that the fans would be a problem.  In the first pic, you can see that the heatsinks overlap the RAM modules, but do not touch then.  Unfortunately, when you insert the fan, it does not clear the RAM modules, and sits too high in the case.

 

i9zJrxol.jpg

 

ZksZXozl.jpg

 

 

So, I thought, "ok...these things are big enough, and my case has plenty of airflow, lets see how they work without the fans installed."  I was shocked at how well they worked without fans.

 

These tests were with both CPUs pegged using PassMark Burn-In.

 

CPU 1 - Supermicro  /  CPU 2 - Noctua (no fan)

r93CPY6.png

 

CPU 1 - Noctua (no fan)  /  CPU 2 - Noctua (no fan)

TwVTc6m.png

 

 

VERY respectable temps, but I wanted to see what I could do to get the fans to fit.  I ended up just notching the bottom corners of the fans with a hot knife.

 

AobHdZTl.jpg

 

 

 

CPU 1 - Noctua (WITH fan)  /  CPU 2 - Noctua (no fan)

yKj7c4m.png

 

So, the CPU with the fan dropped four degrees.  Not critical, but I will probably leave the fans installed.

 

The cables on these Noctua fans are quite short, and I didn't have an extension, so I couldn't install both fans yesterday. I have an extension being delivered today, so I will run one final test with both fans.

 

NOTE:

When plugging any fan other than a SuperMicro fan into a SuperMicro motherboard, you need to adjust the IPMI fan thresholds, otherwise the board will detect the fan as spinning too slow, and it will compensate by spinning all of the other fans up to full speed. You can fix this with the IPMITOOL or IPMIUTIL. 

 

Here are the commands you need for that:

 

To get a list of the sensors:

 

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -c

 

This will give you a list of sensors. You need to find the Snum of the fan you wish to change.

 

ipmiutil ver 2.98
isensor: version 2.98
Connecting to node  10.1.2.126
-- BMC version 3.39, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
0004 | Full    | Temperature     | 01 | CPU1 Temp        | OK   | 30.00 C
0047 | Full    | Temperature     | 02 | CPU2 Temp        | OK   | 38.00 C
008a | Full    | Temperature     | 11 | System Temp      | OK   | 30.00 C
00cd | Full    | Temperature     | 12 | Peripheral Temp  | OK   | 36.00 C
0110 | Full    | Temperature     | 0a | PCH Temp         | OK   | 43.00 C
0153 | Full    | Temperature     | b0 | P1-DIMMA1 TEMP   | OK   | 30.00 C
0196 | Full    | Temperature     | b1 | P1-DIMMA2 TEMP   | OK   | 30.00 C
01d9 | Full    | Temperature     | b4 | P1-DIMMB1 TEMP   | OK   | 30.00 C
021c | Full    | Temperature     | b5 | P1-DIMMB2 TEMP   | OK   | 29.00 C
025f | Full    | Temperature     | b8 | P1-DIMMC1 TEMP   | OK   | 28.00 C
02a2 | Full    | Temperature     | b9 | P1-DIMMC2 TEMP   | OK   | 29.00 C
02e5 | Full    | Temperature     | bc | P1-DIMMD1 TEMP   | OK   | 29.00 C
0328 | Full    | Temperature     | bd | P1-DIMMD2 TEMP   | OK   | 29.00 C
036b | Full    | Temperature     | d0 | P2-DIMME1 TEMP   | OK   | 33.00 C
03ae | Full    | Temperature     | d1 | P2-DIMME2 TEMP   | OK   | 33.00 C
03f1 | Full    | Temperature     | d4 | P2-DIMMF1 TEMP   | OK   | 32.00 C
0434 | Full    | Temperature     | d5 | P2-DIMMF2 TEMP   | OK   | 32.00 C
0477 | Full    | Temperature     | d8 | P2-DIMMG1 TEMP   | OK   | 32.00 C
04ba | Full    | Temperature     | d9 | P2-DIMMG2 TEMP   | OK   | 33.00 C
04fd | Full    | Temperature     | dc | P2-DIMMH1 TEMP   | OK   | 32.00 C
0540 | Full    | Temperature     | dd | P2-DIMMH2 TEMP   | OK   | 32.00 C
0583 | Full    | Fan             | 41 | FAN1             | OK   | 2925.00 RPM
05c6 | Full    | Fan             | 42 | FAN2             | OK   | 2925.00 RPM
0609 | Full    | Fan             | 43 | FAN3             | OK   | 2925.00 RPM
064c | Full    | Fan             | 44 | FAN4             | OK   | 1950.00 RPM
068f | Full    | Fan             | 45 | FAN5             | OK   | 5175.00 RPM
06d2 | Full    | Fan             | 46 | FAN6             | OK   | 5025.00 RPM
0715 | Full    | Fan             | 47 | FANA             | Absent | 0.00 na
0758 | Full    | Fan             | 48 | FANB             | OK   | 2925.00 RPM
079b | Full    | Voltage         | 30 | VTT              | OK   | 1.04 V
07de | Full    | Voltage         | 21 | CPU1 Vcore       | OK   | 0.75 V
0821 | Full    | Voltage         | 22 | CPU2 Vcore       | OK   | 0.75 V
0864 | Full    | Voltage         | 61 | VDIMM AB         | OK   | 1.47 V
08a7 | Full    | Voltage         | 62 | VDIMM CD         | OK   | 1.49 V
08ea | Full    | Voltage         | 63 | VDIMM EF         | OK   | 1.49 V
092d | Full    | Voltage         | 64 | VDIMM GH         | OK   | 1.49 V
0970 | Full    | Voltage         | 31 | +1.1 V           | OK   | 1.09 V
09b3 | Full    | Voltage         | 32 | +1.5 V           | OK   | 1.49 V
09f6 | Full    | Voltage         | 33 | 3.3V             | OK   | 3.31 V
0a39 | Full    | Voltage         | 34 | +3.3VSB          | OK   | 3.41 V
0a7c | Full    | Voltage         | 35 | 5V               | OK   | 4.99 V
0abf | Full    | Voltage         | 36 | +5VSB            | OK   | 4.86 V
0b02 | Full    | Voltage         | 37 | 12V              | OK   | 12.08 V
0b45 | Full    | Voltage         | 38 | VBAT             | OK   | 3.22 V
0b88 | Full    | Drive Slot      | 52 | HDD Status       | Absent | 0.00 na
0bcb | Full    | Platform Securi | 51 | Chassis Intru    | OK   | 0.00 na
0c0e | Compact | Power Supply    | c1 | PS2 Status       | 01 Present |

 

Then, run this to actually set the thresholds.

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -n *snum* -l *lower threshold* -h *upper threshold*

 

I used 200 for the lower threshold and 2500 for the upper threshold.

 

Lastly, you MUST shutdown the server and power it back on for the values to take effect.

Share this post


Link to post

NOTE:

When plugging any fan other than a SuperMicro fan into a SuperMicro motherboard, you need to adjust the IPMI fan thresholds, otherwise the board will detect the fan as spinning too slow, and it will compensate by spinning all of the other fans up to full speed. You can fix this with the IPMITOOL or IPMIUTIL. 

 

Here are the commands you need for that:

 

To get a list of the sensors:

 

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -c

 

This will give you a list of sensors. You need to find the Snum of the fan you wish to change.

 

ipmiutil ver 2.98
isensor: version 2.98
Connecting to node  10.1.2.126
-- BMC version 3.39, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
0004 | Full    | Temperature     | 01 | CPU1 Temp        | OK   | 30.00 C
0583 | Full    | Fan             | 41 | FAN1             | OK   | 2925.00 RPM
05c6 | Full    | Fan             | 42 | FAN2             | OK   | 2925.00 RPM
0609 | Full    | Fan             | 43 | FAN3             | OK   | 2925.00 RPM
064c | Full    | Fan             | 44 | FAN4             | OK   | 1950.00 RPM
068f | Full    | Fan             | 45 | FAN5             | OK   | 5175.00 RPM
06d2 | Full    | Fan             | 46 | FAN6             | OK   | 5025.00 RPM
0715 | Full    | Fan             | 47 | FANA             | Absent | 0.00 na
0758 | Full    | Fan             | 48 | FANB             | OK   | 2925.00 RPM
079b | Full    | Voltage         | 30 | VTT              | OK   | 1.04 V
07de | Full    | Voltage         | 21 | CPU1 Vcore       | OK   | 0.75 V
0821 | Full    | Voltage         | 22 | CPU2 Vcore       | OK   | 0.75 V
0864 | Full    | Voltage         | 61 | VDIMM AB         | OK   | 1.47 V

 

Then, run this to actually set the thresholds.

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -n *snum* -l *lower threshold* -h *upper threshold*

Very neat! :)

 

Does this work with other boards that use IPMI, or only for Supermicro? ie: ASRock?

 

Also, do I need to be doing this through a vm, or can it be done from a local windows box?

 

Finally, is there a way to list the current thresholds (high and low)?

 

Thanks!

Share this post


Link to post

 

Does this work with other boards that use IPMI, or only for Supermicro? ie: ASRock?

 

Also, do I need to be doing this through a vm, or can it be done from a local windows box?

 

Finally, is there a way to list the current thresholds (high and low)?

 

Thanks!

 

I have no clue if you can change other boars, or if it is even necessary.

 

You can do it on any machine that you can run either IPMITOOL or IPMIUTIL.

 

You can see the current thresholds either with IPMITOOL or in the IPMI web interface (FAN4 is the Noctua fan):

 

root@nas:~# ipmitool -I lanplus -H <IP of IPMI> -U <username> -P <PASSWORD> sensor list all
CPU1 Temp        | 39.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 98.000    | 100.000
CPU2 Temp        | 51.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 98.000    | 100.000
System Temp      | 40.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Peripheral Temp  | 42.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
PCH Temp         | 44.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000
P1-DIMMA1 TEMP   | 45.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMA2 TEMP   | 46.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMB1 TEMP   | 45.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMB2 TEMP   | 41.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMC1 TEMP   | 40.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMC2 TEMP   | 42.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMD1 TEMP   | 43.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P1-DIMMD2 TEMP   | 42.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMME1 TEMP   | 46.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMME2 TEMP   | 48.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMF1 TEMP   | 48.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMF2 TEMP   | 47.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMG1 TEMP   | 49.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMG2 TEMP   | 52.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMH1 TEMP   | 52.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
P2-DIMMH2 TEMP   | 50.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
FAN1             | 1875.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN2             | 1950.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN3             | 1875.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN4             | 1200.000   | RPM        | ok    | 75.000    | 150.000   | 225.000   | 2475.000  | 2550.000  | 2625.000
FAN5             | 3750.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FAN6             | 3675.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
FANA             | na         |            | na    | na        | na        | na        | na        | na        | na
FANB             | 1875.000   | RPM        | ok    | 300.000   | 450.000   | 600.000   | 18975.000 | 19050.000 | 19125.000
VTT              | 1.040      | Volts      | ok    | 0.816     | 0.864     | 0.912     | 1.344     | 1.392     | 1.440
CPU1 Vcore       | 0.848      | Volts      | ok    | 0.480     | 0.512     | 0.544     | 1.488     | 1.520     | 1.552
CPU2 Vcore       | 1.008      | Volts      | ok    | 0.480     | 0.512     | 0.544     | 1.488     | 1.520     | 1.552
VDIMM AB         | 1.472      | Volts      | ok    | 1.104     | 1.152     | 1.200     | 1.648     | 1.696     | 1.744
VDIMM CD         | 1.488      | Volts      | ok    | 1.104     | 1.152     | 1.200     | 1.648     | 1.696     | 1.744
VDIMM EF         | 1.488      | Volts      | ok    | 1.104     | 1.152     | 1.200     | 1.648     | 1.696     | 1.744
VDIMM GH         | 1.488      | Volts      | ok    | 1.104     | 1.152     | 1.200     | 1.648     | 1.696     | 1.744
+1.1 V           | 1.088      | Volts      | ok    | 0.880     | 0.928     | 0.976     | 1.216     | 1.264     | 1.312
+1.5 V           | 1.488      | Volts      | ok    | 1.248     | 1.296     | 1.344     | 1.648     | 1.696     | 1.744
3.3V             | 3.312      | Volts      | ok    | 2.640     | 2.784     | 2.928     | 3.648     | 3.792     | 3.936
+3.3VSB          | 3.408      | Volts      | ok    | 2.640     | 2.784     | 2.928     | 3.648     | 3.792     | 3.936
5V               | 4.928      | Volts      | ok    | 4.096     | 4.288     | 4.480     | 5.504     | 5.696     | 6.912
+5VSB            | 4.864      | Volts      | ok    | 4.096     | 4.288     | 4.480     | 5.504     | 5.696     | 6.912
12V              | 12.084     | Volts      | ok    | 10.176    | 10.494    | 10.812    | 13.250    | 13.568    | 13.886
VBAT             | 3.264      | Volts      | ok    | 2.400     | 2.544     | 2.688     | 3.312     | 3.456     | 3.600
HDD Status       | na         | discrete   | na    | na        | na        | na        | na        | na        | na
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
PS2 Status       | 0x1        | discrete   | 0x0100| na        | na        | na        | na        | na        | na

 

CCVlq5J.png

Share this post


Link to post

Noctua certainly knows what they are doing when it comes to cooling!!

 

When I first installed the dual Xeons, I used the SuperMicro SNK-P0048AP4 2U Active DP CPU Heatsinks.  Under full load, they were seriously loud. Even under light loads, they had a high-pitched whine to them.  I can handle fan noise, but these were ridiculous!

 

I purchased a pair of Noctua NH-D9DX i4 3U from Amazon for $54/each. These things are enormous!  But, I when it comes to cooling, that is really what you want.

 

The instructions tell you to remove the fan before installing the heatsink, so I did. The heatsink itself fit just fine. However, I could tell that the fans would be a problem.  In the first pic, you can see that the heatsinks overlap the RAM modules, but do not touch then.  Unfortunately, when you insert the fan, it does not clear the RAM modules, and sits too high in the case.

 

i9zJrxol.jpg

 

ZksZXozl.jpg

 

 

So, I thought, "ok...these things are big enough, and my case has plenty of airflow, lets see how they work without the fans installed."  I was shocked at how well they worked without fans.

 

These tests were with both CPUs pegged using PassMark Burn-In.

 

CPU 1 - Supermicro  /  CPU 2 - Noctua (no fan)

r93CPY6.png

 

CPU 1 - Noctua (no fan)  /  CPU 2 - Noctua (no fan)

TwVTc6m.png

 

 

VERY respectable temps, but I wanted to see what I could do to get the fans to fit.  I ended up just notching the bottom corners of the fans with a hot knife.

 

AobHdZTl.jpg

 

 

 

CPU 1 - Noctua (WITH fan)  /  CPU 2 - Noctua (no fan)

yKj7c4m.png

 

So, the CPU with the fan dropped four degrees.  Not critical, but I will probably leave the fans installed.

 

The cables on these Noctua fans are quite short, and I didn't have an extension, so I couldn't install both fans yesterday. I have an extension being delivered today, so I will run one final test with both fans.

 

NOTE:

When plugging any fan other than a SuperMicro fan into a SuperMicro motherboard, you need to adjust the IPMI fan thresholds, otherwise the board will detect the fan as spinning too slow, and it will compensate by spinning all of the other fans up to full speed. You can fix this with the IPMITOOL or IPMIUTIL. 

 

Here are the commands you need for that:

 

To get a list of the sensors:

 

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -c

 

This will give you a list of sensors. You need to find the Snum of the fan you wish to change.

 

ipmiutil ver 2.98
isensor: version 2.98
Connecting to node  10.1.2.126
-- BMC version 3.39, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
0004 | Full    | Temperature     | 01 | CPU1 Temp        | OK   | 30.00 C
0047 | Full    | Temperature     | 02 | CPU2 Temp        | OK   | 38.00 C
008a | Full    | Temperature     | 11 | System Temp      | OK   | 30.00 C
00cd | Full    | Temperature     | 12 | Peripheral Temp  | OK   | 36.00 C
0110 | Full    | Temperature     | 0a | PCH Temp         | OK   | 43.00 C
0153 | Full    | Temperature     | b0 | P1-DIMMA1 TEMP   | OK   | 30.00 C
0196 | Full    | Temperature     | b1 | P1-DIMMA2 TEMP   | OK   | 30.00 C
01d9 | Full    | Temperature     | b4 | P1-DIMMB1 TEMP   | OK   | 30.00 C
021c | Full    | Temperature     | b5 | P1-DIMMB2 TEMP   | OK   | 29.00 C
025f | Full    | Temperature     | b8 | P1-DIMMC1 TEMP   | OK   | 28.00 C
02a2 | Full    | Temperature     | b9 | P1-DIMMC2 TEMP   | OK   | 29.00 C
02e5 | Full    | Temperature     | bc | P1-DIMMD1 TEMP   | OK   | 29.00 C
0328 | Full    | Temperature     | bd | P1-DIMMD2 TEMP   | OK   | 29.00 C
036b | Full    | Temperature     | d0 | P2-DIMME1 TEMP   | OK   | 33.00 C
03ae | Full    | Temperature     | d1 | P2-DIMME2 TEMP   | OK   | 33.00 C
03f1 | Full    | Temperature     | d4 | P2-DIMMF1 TEMP   | OK   | 32.00 C
0434 | Full    | Temperature     | d5 | P2-DIMMF2 TEMP   | OK   | 32.00 C
0477 | Full    | Temperature     | d8 | P2-DIMMG1 TEMP   | OK   | 32.00 C
04ba | Full    | Temperature     | d9 | P2-DIMMG2 TEMP   | OK   | 33.00 C
04fd | Full    | Temperature     | dc | P2-DIMMH1 TEMP   | OK   | 32.00 C
0540 | Full    | Temperature     | dd | P2-DIMMH2 TEMP   | OK   | 32.00 C
0583 | Full    | Fan             | 41 | FAN1             | OK   | 2925.00 RPM
05c6 | Full    | Fan             | 42 | FAN2             | OK   | 2925.00 RPM
0609 | Full    | Fan             | 43 | FAN3             | OK   | 2925.00 RPM
064c | Full    | Fan             | 44 | FAN4             | OK   | 1950.00 RPM
068f | Full    | Fan             | 45 | FAN5             | OK   | 5175.00 RPM
06d2 | Full    | Fan             | 46 | FAN6             | OK   | 5025.00 RPM
0715 | Full    | Fan             | 47 | FANA             | Absent | 0.00 na
0758 | Full    | Fan             | 48 | FANB             | OK   | 2925.00 RPM
079b | Full    | Voltage         | 30 | VTT              | OK   | 1.04 V
07de | Full    | Voltage         | 21 | CPU1 Vcore       | OK   | 0.75 V
0821 | Full    | Voltage         | 22 | CPU2 Vcore       | OK   | 0.75 V
0864 | Full    | Voltage         | 61 | VDIMM AB         | OK   | 1.47 V
08a7 | Full    | Voltage         | 62 | VDIMM CD         | OK   | 1.49 V
08ea | Full    | Voltage         | 63 | VDIMM EF         | OK   | 1.49 V
092d | Full    | Voltage         | 64 | VDIMM GH         | OK   | 1.49 V
0970 | Full    | Voltage         | 31 | +1.1 V           | OK   | 1.09 V
09b3 | Full    | Voltage         | 32 | +1.5 V           | OK   | 1.49 V
09f6 | Full    | Voltage         | 33 | 3.3V             | OK   | 3.31 V
0a39 | Full    | Voltage         | 34 | +3.3VSB          | OK   | 3.41 V
0a7c | Full    | Voltage         | 35 | 5V               | OK   | 4.99 V
0abf | Full    | Voltage         | 36 | +5VSB            | OK   | 4.86 V
0b02 | Full    | Voltage         | 37 | 12V              | OK   | 12.08 V
0b45 | Full    | Voltage         | 38 | VBAT             | OK   | 3.22 V
0b88 | Full    | Drive Slot      | 52 | HDD Status       | Absent | 0.00 na
0bcb | Full    | Platform Securi | 51 | Chassis Intru    | OK   | 0.00 na
0c0e | Compact | Power Supply    | c1 | PS2 Status       | 01 Present |

 

Then, run this to actually set the thresholds.

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -n *snum* -l *lower threshold* -h *upper threshold*

 

I used 200 for the lower threshold and 2500 for the upper threshold.

 

Lastly, you MUST shutdown the server and power it back on for the values to take effect.

If you have a 4U case the NH-U9DX i4 doesn't have the same problem with the RAM modules. And it comes with two fans.

I have both as I was too impatient to order another NH-U9DX i4 and bought the 3u version locally.

Share this post


Link to post

 

If you have a 4U case the NH-U9DX i4 doesn't have the same problem with the RAM modules. And it comes with two fans.

I have both as I was too impatient to order another NH-U9DX i4 and bought the 3u version locally.

 

I do not have a 4U case.

Share this post


Link to post

I purchased the same chassis and air duct.  I also bought the 8x3.5 tray separately for  $40 and swapped out the 2 2.5 drive trays.  I agree with the hairdryer description of the noise.  I've only booted into the bios so far but setting the acoustic levels hasn't improved things.  My 2600cp is on an ancient bios so I'm hoping that will make a difference.  If not I'll swap out fans and put a different powers supply in .

 

Running through the BIOS update (and the subsequent FRUSDRUPDATE) *will* reconfigure the BMC for your chassis configuration and allow the BMC to properly control the cooling domains and substantially slow the fans down.  This is mandatory for proper operation.

 

After the update fan noise is now very low when idling  ill do some more testing next week to see how it does under load after I receive the stock heatsink that match the air duct. If things hold up these intel chasses with powersupply will prove to be a fantastic bargin.  under $230 including shipping for case, redundant 750 ps, 8x3.5 hot swap drives, 2x heat sinks, air duct  and 2 extra 8x2.5 hot swap bays.  I don't see how you can beat it.  I'm debating building a second one.

 

Brian

Share this post


Link to post

 

If you have a 4U case the NH-U9DX i4 doesn't have the same problem with the RAM modules. And it comes with two fans.

I have both as I was too impatient to order another NH-U9DX i4 and bought the 3u version locally.

 

I do not have a 4U case.

Can't you hot rod mod your case?  ;D

Share this post


Link to post

ICcxHNV.jpg

 

Well finally got 4 cards to fit, wound up being a total ghetto rigged hack job but oh well, they all still work!

 

 

Details, build log

Share this post


Link to post

So I have the Nanoxia Deep Silence 6 tower and the Intel S2600CP. Where do I connect the Power LED +/- and Power Switch ends? I found where the HD LED goes but can't figure out where the others go? Please help! I am almost done with my build.

Share this post


Link to post

Not sure if I found the correct PINS. I placed PWR BTN on PIn 11/13 on the Front Panel Header PIN-Out and HDD LED on 7/9 and PWR LED on 5/6. I've attached the screenshot of the Front Panel Header layout within the Intel S2600CP document.

 

Not sure if this is the correct place for the PIN Outs coming from my case. Also getting Beep Code 1-5-4-4 and can't seem to figure out why. I tried resetting BIOS by removing the battery but it continues to beep after I reconnect to power.

 

Any suggestions?

Capture.JPG.27d1ce578d9d1e42ec56104ec3307543.JPG

Share this post


Link to post

I removed all SAS card (2), removed the RMM4LITE NIC, disconnected two SATA power cables from PSU and then reconnected to Power. No more Beeping but it still will not power on. Although even before I removed everything mentioned I was able to access the BMC console via the IP address. When I first reconnect it to Power and/or Press the Power Button it attempts to power up and I see the fans spin briefly and also hear a sound from the internal speaker and then it stops. I see Green lights on both the Front and Back LED's but still am unable to power up.

 

Any suggestions? I will soon attempt to disconnect the motherboard and make sure it can run outside of the case in the event one of the motherboard mounting points is causing this issue.

Share this post


Link to post

Tested outside of case with a 650W PSU and it is doing the same thing. No beep just the CPU fans spinning initially when power is introduced and also the sound from the onboard speaker (sounds almost like a cricket) and then fans stop. LEDs remain on with initial Blue LED flashing and then solid Green LEDs Front and Back. Only things left on board are Memory, Processors and Noctua Heatsink and Fans.

 

Memory is in Blue slots (A1, B1 C1, etc) nothing is secondary memory slots. Only have 64GB RAM.

Share this post


Link to post

I'd go down to 1 CPU and two DIMMS of memory to try and narrow down if it's a bad CPU / Memory

Share this post


Link to post

So I have the Nanoxia Deep Silence 6 tower and the Intel S2600CP. Where do I connect the Power LED +/- and Power Switch ends? I found where the HD LED goes but can't figure out where the others go? Please help! I am almost done with my build.

 

Page 99 of the s2600cp family tech prod spec document shows where to hook the Power LED, HD LED, Power button, Reset Button, and NIC LEDs:

http://www.intel.com/content/dam/support/us/en/documents/motherboards/server/s2600cp/sb/g26942005_s2600cp_p4000cp_tps_rev19.pdf

 

...on page 104 are descriptions of the onboard diagnostic LEDs to help t-shoot board level issues.  There are POST code LEDs.

 

Page 201 Post Code Diagnostic LED Decoder...  Followed by on page 206 the Post Code Error table.

 

Beep code 1-5-4-4 is Power Control Fault per the Beep Code table on page 211.

 

 

 

Share this post


Link to post

Looking to do a build and came across this thread.... Have a question or two....

 

About the only single CPU ATX board I can find for these is the (happy for other recommendations)

http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm

 

No PCIe x16 though (x16 slots but x8). Probably not an issue as I'd only be thinking of running a HTPC off one of the slots but would like a Nvidia 960 in there. Is there much of an issue running that card in an x8 slot? I'll want at least 3 cards in there but I should be ok with some x8 cards in the other slots.

 

I was looking to buy the E5-2620v3. Is there much real world difference in the idle power draw of these two chips?

 

I'll go with the cloudninjas RAM.

 

You probably want this one if you get a v1 or v2 chip:

http://www.asrockrack.com/general/productdetail.asp?Model=EPC602D8A#Specifications

 

2 x PCIE3.0 x16, 4 x PCIE3.0 x8, 1 x PCIE2.0 x8

 

Thanks, that looks like a good board. So the EPC602D8A, this E5-2670 and the Cloudninjas RAM could be a good combo. Works out cheaper than the 2620v3 combo I was looking at.

 

The question is though..... When comparing the two, would the 2620v3 be the better CPU for a 24/7 machine taking into account it won't be under that much pressure (close to idle a lot of the time)

2620v3 - 6/12 core/thread, 2.4GHz 85w

2670 - 8/16, 2.6GHz, 115w

Share this post


Link to post

I'd go down to 1 CPU and two DIMMS of memory to try and narrow down if it's a bad CPU / Memory

 

I went down to one CPU and one memory after trying your suggestion and same results. As soon as I flick on the power manually from the PSU I see the Green and Blue LED as well as the Front Green LED. I hear the speaker make a single "Click" sound. The rear Blue and Green LED blink steadily and then stop and only the Green LED remains. After it settles, roughly 20-30 seconds the PSU fan, CPU fan turn on briefly alongside the same "Click" from the speaker. Happens two times like that and then nothing although Green LEDs are on.

 

I am able to access the BMC console when the board is connected to my network.

Share this post


Link to post

 

NOTE:

When plugging any fan other than a SuperMicro fan into a SuperMicro motherboard, you need to adjust the IPMI fan thresholds, otherwise the board will detect the fan as spinning too slow, and it will compensate by spinning all of the other fans up to full speed. You can fix this with the IPMITOOL or IPMIUTIL. 

 

Here are the commands you need for that:

 

To get a list of the sensors:

 

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -c

 

This will give you a list of sensors. You need to find the Snum of the fan you wish to change.

 

ipmiutil ver 2.98
isensor: version 2.98
Connecting to node  10.1.2.126
-- BMC version 3.39, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
0004 | Full    | Temperature     | 01 | CPU1 Temp        | OK   | 30.00 C
0047 | Full    | Temperature     | 02 | CPU2 Temp        | OK   | 38.00 C
008a | Full    | Temperature     | 11 | System Temp      | OK   | 30.00 C
00cd | Full    | Temperature     | 12 | Peripheral Temp  | OK   | 36.00 C
0110 | Full    | Temperature     | 0a | PCH Temp         | OK   | 43.00 C
0153 | Full    | Temperature     | b0 | P1-DIMMA1 TEMP   | OK   | 30.00 C
0196 | Full    | Temperature     | b1 | P1-DIMMA2 TEMP   | OK   | 30.00 C
01d9 | Full    | Temperature     | b4 | P1-DIMMB1 TEMP   | OK   | 30.00 C
021c | Full    | Temperature     | b5 | P1-DIMMB2 TEMP   | OK   | 29.00 C
025f | Full    | Temperature     | b8 | P1-DIMMC1 TEMP   | OK   | 28.00 C
02a2 | Full    | Temperature     | b9 | P1-DIMMC2 TEMP   | OK   | 29.00 C
02e5 | Full    | Temperature     | bc | P1-DIMMD1 TEMP   | OK   | 29.00 C
0328 | Full    | Temperature     | bd | P1-DIMMD2 TEMP   | OK   | 29.00 C
036b | Full    | Temperature     | d0 | P2-DIMME1 TEMP   | OK   | 33.00 C
03ae | Full    | Temperature     | d1 | P2-DIMME2 TEMP   | OK   | 33.00 C
03f1 | Full    | Temperature     | d4 | P2-DIMMF1 TEMP   | OK   | 32.00 C
0434 | Full    | Temperature     | d5 | P2-DIMMF2 TEMP   | OK   | 32.00 C
0477 | Full    | Temperature     | d8 | P2-DIMMG1 TEMP   | OK   | 32.00 C
04ba | Full    | Temperature     | d9 | P2-DIMMG2 TEMP   | OK   | 33.00 C
04fd | Full    | Temperature     | dc | P2-DIMMH1 TEMP   | OK   | 32.00 C
0540 | Full    | Temperature     | dd | P2-DIMMH2 TEMP   | OK   | 32.00 C
0583 | Full    | Fan             | 41 | FAN1             | OK   | 2925.00 RPM
05c6 | Full    | Fan             | 42 | FAN2             | OK   | 2925.00 RPM
0609 | Full    | Fan             | 43 | FAN3             | OK   | 2925.00 RPM
064c | Full    | Fan             | 44 | FAN4             | OK   | 1950.00 RPM
068f | Full    | Fan             | 45 | FAN5             | OK   | 5175.00 RPM
06d2 | Full    | Fan             | 46 | FAN6             | OK   | 5025.00 RPM
0715 | Full    | Fan             | 47 | FANA             | Absent | 0.00 na
0758 | Full    | Fan             | 48 | FANB             | OK   | 2925.00 RPM
079b | Full    | Voltage         | 30 | VTT              | OK   | 1.04 V
07de | Full    | Voltage         | 21 | CPU1 Vcore       | OK   | 0.75 V
0821 | Full    | Voltage         | 22 | CPU2 Vcore       | OK   | 0.75 V
0864 | Full    | Voltage         | 61 | VDIMM AB         | OK   | 1.47 V
08a7 | Full    | Voltage         | 62 | VDIMM CD         | OK   | 1.49 V
08ea | Full    | Voltage         | 63 | VDIMM EF         | OK   | 1.49 V
092d | Full    | Voltage         | 64 | VDIMM GH         | OK   | 1.49 V
0970 | Full    | Voltage         | 31 | +1.1 V           | OK   | 1.09 V
09b3 | Full    | Voltage         | 32 | +1.5 V           | OK   | 1.49 V
09f6 | Full    | Voltage         | 33 | 3.3V             | OK   | 3.31 V
0a39 | Full    | Voltage         | 34 | +3.3VSB          | OK   | 3.41 V
0a7c | Full    | Voltage         | 35 | 5V               | OK   | 4.99 V
0abf | Full    | Voltage         | 36 | +5VSB            | OK   | 4.86 V
0b02 | Full    | Voltage         | 37 | 12V              | OK   | 12.08 V
0b45 | Full    | Voltage         | 38 | VBAT             | OK   | 3.22 V
0b88 | Full    | Drive Slot      | 52 | HDD Status       | Absent | 0.00 na
0bcb | Full    | Platform Securi | 51 | Chassis Intru    | OK   | 0.00 na
0c0e | Compact | Power Supply    | c1 | PS2 Status       | 01 Present |

 

Then, run this to actually set the thresholds.

ipmiutil.exe sensor -N *IP address of IPMI interface* -U *Username* -P *Password* -n *snum* -l *lower threshold* -h *upper threshold*

 

I used 200 for the lower threshold and 2500 for the upper threshold.

 

Lastly, you MUST shutdown the server and power it back on for the values to take effect.

 

A big thank you for the info on how to change the thresholds for the fans. Now it's quiet :)

Share this post


Link to post

Just tried powering on with none of the memory slots occupied. I would have thought I'd get a series of beeps but all i got was the same symptoms. Click, LED's and spinning of fans for a brief 2 seconds. Shouldn't the board alert me that there is no memory installed? Going to try without the CPU and see if I get beeps. Starting to feel like this is a faulty board.

Share this post


Link to post

Tried it without memory and CPU and finally got a 1-5-2-1 beep error. I swapped CPU's and placed the CPU that was previously in CPU socket 2 in CPU socket 1 and placed two different memory sticks in A1 and B1. Same results with Click, spin and LED's. Also same Event Log results.

 

Not sure what it is and I am beginning to think its time to shop for a new board. Any suggestions would be helpful whether for a new board or with what you think could be the issue.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.