$90 Xeon E5-2670 2.6Ghz (8cores / 16threads)

[jwegman]

Running through the BIOS update (and the subsequent FRUSDRUPDATE) *will* reconfigure the BMC for your chassis configuration and allow the BMC to properly control the cooling domains and substantially slow the fans down.  This is mandatory for proper operation.

Hey Jake

 

Has anybody documented this firmware update yet on the Natex Intel 2600 mb?  Here are the files I have downloaded

 

https://downloadcenter.intel.com/download/25746/Intel-Server-Board-S2600CP-Firmware-Update-Package-for-EFI

 

https://downloadcenter.intel.com/download/23883/S2600-BIOS-and-Firmware-Update-Configuration-Utilities-for-Windows-Linux-WinPE-UEFI-?v=t

 

Just want to make sure I am heading in the right direction before I brick something....

 

How did you do your update?  From the unraid prompt?  (Looks like you don't do this from the bios)

 

Yes, that's the version that I'm using on both my S2600CP2J boards from natex.  Note that I've *only* used the UEFI package version.  This has been talked about several times already in this thread.  Simply unzip it to a FAT formatted USB stick, then boot to the UEFI shell.  The shell should automatically mount the USB stick and find the Startup file to automatically begin the process.  If you happen to put the files in a subdirectory of the USB stick, then you'll have to manually navigate into that directory and kick off the process (per the instructions; give them a read).

 

[tr0910]

Yes, that's the version that I'm using on both my S2600CP2J boards from natex.  Note that I've *only* used the UEFI package version.  This has been talked about several times already in this thread.  Simply unzip it to a FAT formatted USB stick, then boot to the UEFI shell.  The shell should automatically mount the USB stick and find the Startup file to automatically begin the process.  If you happen to put the files in a subdirectory of the USB stick, then you'll have to manually navigate into that directory and kick off the process (per the instructions; give them a read).

 

Thanks Jake, I am up to 4 of those motherboards now, but this is the first p4000 install and the first I need to mess with the bios.  I want to get the noise under control. 

 

My web guy really likes these systems.

 

 

[tr0910]

Yes, that's the version that I'm using on both my S2600CP2J boards from natex. 

 

Partial success.  I have run the update several times and now get the fans to slow down, but I am not sure that I have expected behavior down near the bottom of the update.

 

It wants user input whether to update

 

1. SDR

2. FRU

3. Both SDF and FRU

 

I select both

 

Then it complains fan config not valid. 

 

This forces me to select redundant or non redundant p4000 fan.

 

Mine are the 2x120 non redundant.  So I select that.

 

Then it finds psu1 and psu2 but fails to detect the hotswap backplane.

 

I have the 8x3.5 installed after pulling the 16x2.5 out.  Using 1 mini SAS(HSBP) I have 4 motherboard sata ports connected and one little 5 pin connecter on the same cable connected to the mb near the sata ports.  The other cable for the top set of 4 x 3.5 drives is not connected.

 

There is another little cable in the duct package called sgpio(hsbp/mb) that I am not using.  Spare parts or desperately needed??  <grin>

 

Thanks, Tim

[ijuarez]

any one got a lead on the I/O shield for the natex board?

 

http://www.ebay.com/itm/181527808860?_trksid=p2060353.m1438.l2649&ssPageName=STRK%3AMEBIDX%3AIT

 

I offered $4 and it was accepted

 

thank you, i sent my offer

[jwegman]

Yes, that's the version that I'm using on both my S2600CP2J boards from natex. 

 

Partial success.  I have run the update several times and now get the fans to slow down, but I am not sure that I have expected behavior down near the bottom of the update.

 

It wants user input whether to update

 

1. SDR

2. FRU

3. Both SDF and FRU

 

I select both

 

Then it complains fan config not valid. 

 

This forces me to select redundant or non redundant p4000 fan.

 

Mine are the 2x120 non redundant.  So I select that.

 

Then it finds psu1 and psu2 but fails to detect the hotswap backplane.

 

I have the 8x3.5 installed after pulling the 16x2.5 out.  Using 1 mini SAS(HSBP) I have 4 motherboard sata ports connected and one little 5 pin connecter on the same cable connected to the mb near the sata ports.  The other cable for the top set of 4 x 3.5 drives is not connected.

 

There is another little cable in the duct package called sgpio(hsbp/mb) that I am not using.  Spare parts or desperately needed??  <grin>

 

Thanks, Tim

 

For the Update to properly detect your 2x120 fixed fan configuration, you need those fans plugged into the correct fan headers (sys1 and sys2; they are color coded for the correct fans).  It then should automatically detect the 2 fixed fan config...  Don't plug any other fans into any of the other fan headers during the SDRUpdate.

 

For the update to detect the HSBP, you *need* the I2C cable from the HSBP to the I2C header on the motherboard.

 

Also, out of curiosity, when you installed the motherboard, did you stick on the little rubber pad supports to the bottom of the board (there are little white circles silk screened marking the proper spots)?  These should *not* be overlooked if you want to properly support the board.

 

[grimlock49]

thanks for the post guys my build so far

 

board    https://www.superbiiz.com/detail.php?name=MB-X9SRLFB

ram      http://www.ebay.com/itm/152033507959?_trksid=p2060353.m2749.l2649&ssPageName=STRK%3AMEBIDX%3AIT

cpu      http://www.ebay.com/itm/281889330127?_trksid=p2060353.m2749.l2649&ssPageName=STRK%3AMEBIDX%3AIT

 

the rest I'm recycling from my current server

 

my current case is an Lian Li Armorsuit PC-P80

 

will this cooler fit http://www.amazon.com/Noctua-Cooler-LGA2011-Platforms-NH-U12DXi4/dp/B00DWFQ42I    ??

[tr0910]

For the Update to properly detect your 2x120 fixed fan configuration, you need those fans plugged into the correct fan headers (sys1 and sys2; they are color coded for the correct fans).  It then should automatically detect the 2 fixed fan config...  Don't plug any other fans into any of the other fan headers during the SDRUpdate.

 

For the update to detect the HSBP, you *need* the I2C cable from the HSBP to the I2C header on the motherboard.

 

With the fans connected to the correct ports, it finally worked and I did find in my spare parts a discarded i2c cable.  However, the fans run quieter when connected to the connectors closer to CPU1.  I moved them back there.  Some have mentioned further tuning for noise.  Is that done via bios or via ipmi?  I can't find these options in either place.

 

Also, out of curiosity, when you installed the motherboard, did you stick on the little rubber pad supports to the bottom of the board (there are little white circles silk screened marking the proper spots)?  These should *not* be overlooked if you want to properly support the board.

 

Jake, you surely don't expect me to take it all apart again??  (mumble, grumble)

[jwegman]

For the Update to properly detect your 2x120 fixed fan configuration, you need those fans plugged into the correct fan headers (sys1 and sys2; they are color coded for the correct fans).  It then should automatically detect the 2 fixed fan config...  Don't plug any other fans into any of the other fan headers during the SDRUpdate.

 

For the update to detect the HSBP, you *need* the I2C cable from the HSBP to the I2C header on the motherboard.

 

With the fans connected to the correct ports, it finally worked and I did find in my spare parts a discarded i2c cable.  However, the fans run quieter when connected to the connectors closer to CPU1.  I moved them back there.  Some have mentioned further tuning for noise.  Is that done via bios or via ipmi?  I can't find these options in either place.

 

Also, out of curiosity, when you installed the motherboard, did you stick on the little rubber pad supports to the bottom of the board (there are little white circles silk screened marking the proper spots)?  These should *not* be overlooked if you want to properly support the board.

 

Jake, you surely don't expect me to take it all apart again??  (mumble, grumble)

 

...Honestly, I expected people to at least read the S2600CP family Quick Start Guide

 

Step 3 shows the attachment of the rubber support bumpers:

http://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/g29357003_s2600cp_qsug_front.pdf

and

http://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/g29357003_s2600cp_qsug_back.pdf

[Naldinho]

I think I got everything setup. I can get to Tower Login with a monitor attached but still can't get to tower headless.

 

I have a question about a blinking amber light on the Natex board -- what does that mean?

 

I downloaded the manual for the S2600CP2 but all it says is

 

The server system contains the following diagnostic LEDs, each providing the following functions:

A - The System Status LED on the front and back panels shows the overall health of the system (green,

blinking green, blinking amber, amber, off).

 

I assume Green is what I want but I have blinking amber but there is no discussion of what blinking amber means in the rest of the manual.

 

---

 

I started running the memtest and it said my CPU temp was at 99C  That seems high so I turned everything off.

 

Hopefully someone here will give me some direction.

[jwegman]

I think I got everything setup. I can get to Tower Login with a monitor attached but still can't get to tower headless.

 

I have a question about a blinking amber light on the Natex board -- what does that mean?

 

I downloaded the manual for the S2600CP2 but all it says is

 

The server system contains the following diagnostic LEDs, each providing the following functions:

A - The System Status LED on the front and back panels shows the overall health of the system (green,

blinking green, blinking amber, amber, off).

 

I assume Green is what I want but I have blinking amber but there is no discussion of what blinking amber means in the rest of the manual.

 

If you haven't done so yet, I'd advise that you check/enable the NICs in the BIOS settings.  If I recall correctly, they are disabled by default.

[Naldinho]

I think I got everything setup. I can get to Tower Login with a monitor attached but still can't get to tower headless.

 

I have a question about a blinking amber light on the Natex board -- what does that mean?

 

I downloaded the manual for the S2600CP2 but all it says is

 

The server system contains the following diagnostic LEDs, each providing the following functions:

A - The System Status LED on the front and back panels shows the overall health of the system (green,

blinking green, blinking amber, amber, off).

 

I assume Green is what I want but I have blinking amber but there is no discussion of what blinking amber means in the rest of the manual.

 

If you haven't done so yet, I'd advise that you check/enable the NICs in the BIOS settings.  If I recall correctly, they are disabled by default.

 

I'll check that next time I boot but I've shut it down until I get some clarification on temperature of CPU.

 

I ran the memtest that is on the Unraid boot and temp started at 98C and went to 100 -- when I googled it says 81.8 is max.

 

The heatseaks are hot to the touch -- enough that it is uncomfortable but I think that is expected.

 

---

 

Thank you -- they were disabled. I have enabled them. I assume once I figure out the heat issue I'll be ready to go.

[Naldinho]

I have never seen a BIOS that doesn't give CPU temp but I can't find it on the Aptio Bios that came with the Natex combo. I've been looking for it a while and I even found screen shots that show a Monitor top menu that I don't have -- does anyone who bought the Natex combo know how to get CPU temps from the BIOS?

 

Also is anyone running the combo as it came with just the heat sinks?

[jwegman]

I have never seen a BIOS that doesn't give CPU temp but I can't find it on the Aptio Bios that came with the Natex combo. I've been looking for it a while and I even found screen shots that show a Monitor top menu that I don't have -- does anyone who bought the Natex combo know how to get CPU temps from the BIOS?

 

Also is anyone running the combo as it came with just the heat sinks?

 

Are you talking about those short little 1u passive heat sinks?  I wouldn't consider using them unless you had a proper corresponding chassis, cooling fans, and airduct that would tightly mate with those heat sinks to force *all* the air through them.

 

The BMC webconsole's sensors page will provide temps (not CPU temps directly), however you can derive the CPU temp from the P# Therm Margin values.  Note, if they drop below ~20%, that indicates that your CPU cores are very close to 80c.  The stock Intel cooling setup for that board will ramp up the cooling fans so that a system will stay above the 20% P# Therm Margin.

 

 

[tr0910]

I have never seen a BIOS that doesn't give CPU temp but I can't find it on the Aptio Bios that came with the Natex combo. I've been looking for it a while and I even found screen shots that show a Monitor top menu that I don't have -- does anyone who bought the Natex combo know how to get CPU temps from the BIOS?

 

Also is anyone running the combo as it came with just the heat sinks?

 

Are you talking about those short little 1u passive heat sinks?  I wouldn't consider using them unless you had a proper corresponding chassis, cooling fans, and airduct that would tightly mate with those heat sinks to force *all* the air through them.

 

The BMC webconsole's sensors page will provide temps (not CPU temps directly), however you can derive the CPU temp from the P# Therm Margin values.  Note, if they drop below ~20%, that indicates that your CPU cores are very close to 80c.  The stock Intel cooling setup for that board will ramp up the cooling fans so that a system will stay above the 20% P# Therm Margin.

 

As part of my burnin test for each Natex board, I run memtest, and if I do it with only the passive heat sinks on the Natex Intel 2600 boards, and without a fan, the CPU temps will hit 100 degrees C by which time the thermal throttling is working very well.  While some would shudder at this, I would rather have a failure on the test bench than in production.  I can say that these boards and the Intel 2670's seem to be very resilient.

 

On one of my systems, it is not in a p4000 case, and I am laying a fan on top of passive coolers and that one the Processor Thermal margin runs in the high 50's.  This works fine, but like Jake I can tell you that without some airflow these babies get hot. 

 

[tr0910]

Step 3 shows the attachment of the rubber support bumpers:

http://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/g29357003_s2600cp_qsug_front.pdf

 

I had gotten part way through the 260 page novel, but had completely missed these quick start guides.  Thanks for the link. 

 

I now have the p4000 acoustically acceptable for a server, but the whine of the fans in the 750w PSU is still a little obnoxious.  Were you able to get them to idle down at all??  They seem to be running at full speed still.

[jwegman]

Step 3 shows the attachment of the rubber support bumpers:

http://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/g29357003_s2600cp_qsug_front.pdf

 

I had gotten part way through the 260 page novel, but had completely missed these quick start guides.  Thanks for the link. 

 

I now have the p4000 acoustically acceptable for a server, but the whine of the fans in the 750w PSU is still a little obnoxious.  Were you able to get them to idle down at all??  They seem to be running at full speed still.

 

Do you have the PMBus lead from the PDB (Power Distribution Board) plugged into the motherboard when you ran through the BIOS updates?  If so, did it properly detect that you had the CRPS (common redundant power supplies)?

 

If the BMC was properly configured per what I just stated above; look at the BMC sensor information for anything out of whack, perhaps concerning the PS temps.  Also, any errors in the SEL (system event log)?

 

Also check to make sure that you have both the PS's fully seated.

[Naldinho]

Thank you both for the responses -- good to know this is normal and that I just need to add more cooling.

 

I went with a tower style case from Phanteks -- it comes with a 200mm (front) and 140mm (rear) and the option to add top and bottom fans. It looks like adding 3 top fans is easiest so I'll do that.

 

I don't expect top fans will be sufficient so I've been looking at CPU coolers. Would something either of these be appropriate?

 

Cooler Master Hyper 212 EVO Direct Heat Pipe CPU Cooler R2 http://www.canadacomputers.com/product_info.php?cPath=8_129&item_id=044952

 

Cooler Master Hyper T4 Heatpipe 120mm PWM Fan http://www.canadacomputers.com/product_info.php?cPath=8_129&item_id=057457

 

I have more options if I go online but I was hoping to just buy something at a B&M tomorrow.

 

Noise is a factor. I'd like to keep to keep this reasonably quite

 

On one of my systems, it is not in a p4000 case, and I am laying a fan on top of passive coolers and that one the Processor Thermal margin runs in the high 50's.  This works fine, but like Jake I can tell you that without some airflow these babies get hot.

 

This as a solution would be preferable -- I'd prefer to not have to take everything apart to remove the motherboard or mess around with thermal glue -- never done it before. If I could just add fans to the current heat sinks I would rather do that. I'm not at the office so don't have the server to reference -- I assume the fan just screw into the top of the screws that hold the current heatsinks?

 

If I was to stop at a B&M store what would I ask for (size + anything else I'd need to specify)?

[tr0910]

I just laid a 120mm fan on top of both, straddling both coolers. Since mine is a horizontal rack mount it won't fall off. If you have a vertical tower this isn't so easy.

[Naldinho]

Thanks -- yeah it is a vertical tower so that wouldn't work.

 

Been reading about airlfow and it seems there is more to it then just slap more fans on so doesn't look like I'll have my build up and running today.

[tr0910]

Do you have the PMBus lead from the PDB (Power Distribution Board) plugged into the motherboard when you ran through the BIOS updates?  If so, did it properly detect that you had the CRPS (common redundant power supplies)?

 

If the BMC was properly configured per what I just stated above; look at the BMC sensor information for anything out of whack, perhaps concerning the PS temps.  Also, any errors in the SEL (system event log)?

 

Also check to make sure that you have both the PS's fully seated.

 

After several bios updates, I noticed that cable and yes now the CPRS is noted.  I replugged the PSU back in, and the sensor info looks ok for PSU temps.  I've attached the SEL just in case I missed something.

 

You seem to know these systems very well.  Did you manage a fleet at work?

p4000sel.txt

[jwegman]

Do you have the PMBus lead from the PDB (Power Distribution Board) plugged into the motherboard when you ran through the BIOS updates?  If so, did it properly detect that you had the CRPS (common redundant power supplies)?

 

If the BMC was properly configured per what I just stated above; look at the BMC sensor information for anything out of whack, perhaps concerning the PS temps.  Also, any errors in the SEL (system event log)?

 

Also check to make sure that you have both the PS's fully seated.

 

After several bios updates, I noticed that cable and yes now the CPRS is noted.  I replugged the PSU back in, and the sensor info looks ok for PSU temps.  I've attached the SEL just in case I missed something.

 

You seem to know these systems very well.  Did you manage a fleet at work?

 

In your SEL, I see a bunch of asserted SYSFAN 1 and 2 errors.

 

Please understand that for the BMC to properly monitor and maintain the temps of the Cooling Domains, those two 120mm fans *need* to be plugged into SYSFAN 1 and 2 (make sure that the blue fan lead mates up with the SYSFAN with the blue header).

 

The BMC uses various sensors in the respective cooling domains to ramp up the fans assigned to that domain via a closed loop system.  That won't happen when things are not hooked up correctly.  There are two cooling domains, the PCI cards, and the CPUs.  The lower 120mm fan is dedicated to the PCI domain, the top to the CPU domain.

 

You mentioned previously that the 2x120mm fans were loud when connected to SYSFAN 1 and 2; how fast where they spinning (per the sensor page on the BMC web console)?  Did you have the top case cover on?  When the chassis intrusion detection switch sensor is asserted, then it'll automatically drive all the fans at 100%.  Did you check the BIOS setting Fan PWM offset?  Make sure that it's 0 otherwise it'll skew the fan idle speeds.  Make sure that your Altitude is set to reflect where you actually are.  Page 70 of the S2600 & P4000 Technical Spec document lists all the various physical and virtual sensors uses to control the fans.

 

The tech spec document also has a neato diagram on page 71 that shows all the things that can result in fan speed changes...

[tr0910]

The BMC uses various sensors in the respective cooling domains to ramp up the fans assigned to that domain via a closed loop system.  That won't happen when things are not hooked up correctly.  There are two cooling domains, the PCI cards, and the CPUs.  The lower 120mm fan is dedicated to the PCI domain, the top to the CPU domain.

 

A W E S O M E . . . now it has gone dead silent.  48 watts on PSU1 that is unbelievable ! ! !

 

The case was open before when I didn't like the sys1 and sys2 fan noise.  This p4000 with Intel 2600cp2 combo is brilliant, but you have to get it wired and updated properly.  Thanks Jake, for the tips.

 

Now I still show an issue with system fan 2, but since I have no pci cards installed, maybe it was smart enough to shut it down.

 

Finally, the big question.  One of my other 2600 based systems isn't in a p4000, and doesn't have this quality of PSU but has dual processors and 64gb RAM, and is pulling about 200 watts at idle.  I think I want to it it in a p4000 just like this.  I like the 48 watt idle on this test system (and it is confirmed by the reading on my APC UPS with one 2670 cpu, a 7200 rpm Seagate 3TB and 32gb RAM.)

 

What is the lowest power draw you have seen with dual cpu's installed??  Can we get the 2nd cpu to go into deep sleep mode and only wake up when there is serious work to do?

 

I am using Xenserver 6.5 and Linux VM's

 

Sensor Readings below...
Pwr Unit Status   All deasserted   OK   0x0000
Pwr Unit Redund   reports full redundancy has been regained    OK   0x0001
IPMI Watchdog   All deasserted   OK   0x0000
Physical Scrty   All deasserted   OK   0x0000
FP NMI Diag Int   All deasserted   OK   0x0000
SMI Timeout   All deasserted   OK   0x0000
System Event Log   All deasserted   OK   0x0000
System Event   All deasserted   OK   0x0000
Button   All deasserted   OK   0x0000
VR Watchdog   All deasserted   OK   0x0000
SSB Therm Trip   All deasserted   OK   0x0000
BMC FW Health   All deasserted   OK   0x0000
System Airflow   Normal    OK   70 CFM
BB EDGE Temp   Normal    OK   23 degrees C
Front Panel Temp   Normal    OK   24 degrees C
SSB Temp   Normal    OK   43 degrees C
BB BMC Temp   Normal    OK   39 degrees C
BB P2 VR Temp   Normal    OK   23 degrees C
BB MEM VR Temp   Normal    OK   27 degrees C
HSBP 1 Temp   Normal    OK   24 degrees C
Exit Air Temp   Normal    OK   24 degrees C
LAN NIC Temp   Normal    OK   53 degrees C
System Fan 1   Normal    OK   1092 RPM
System Fan 2   All deasserted   Unknown   Not Available
PS1 Status   reports the power supply's presence has been detected    OK   0x0001
PS2 Status   reports the power supply's presence has been detected    OK   0x0001
PS1 Power In   Normal    OK   48 Watts
PS2 Power In   Normal    OK   4 Watts
PS1 Curr Out %   Normal    OK   4
PS2 Curr Out %   Normal    OK   0
PS1 Temperature   Normal    OK   27 degrees C
PS2 Temperature   Normal    OK   33 degrees C
HSC1 Status   All deasserted   OK   0x0000
P1 Status   reports the processor's presence has been detected    OK   0x0080
P1 Therm Margin   Normal    OK   -55 degrees C
P1 Therm Ctrl %   Normal    OK   0
P1 ERR2   All deasserted   OK   0x0000
CATERR   All deasserted   OK   0x0000
P1 MSID Mismatch   All deasserted   OK   0x0000
CPU Missing   All deasserted   OK   0x0000
P1 DTS Therm Mgn   Normal    OK   -55 degrees C
P1 VRD Hot   All deasserted   OK   0x0000
P1 MEM01 VRD Hot   All deasserted   OK   0x0000
P1 MEM23 VRD Hot   All deasserted   OK   0x0000
PS1 Fan Fail   All deasserted   OK   0x0000
PS2 Fan Fail   All deasserted   OK   0x0000
DIMM Thrm Mrgn 1   Normal    OK   -51 degrees C
DIMM Thrm Mrgn 2   Normal    OK   -51 degrees C
Mem P1 Thrm Trip   All deasserted   OK   0x0000
Agg Thrm Mgn 1   Normal    OK   -30 degrees C
Agg Thrm Mgn 2   Normal    OK   -26 degrees C
BB +12.0V   Normal    OK   11.935 Volts
BB +5.0V   Normal    OK   5.002 Volts
BB +3.3V   Normal    OK   3.382 Volts
BB +5.0V STBY   Normal    OK   4.915 Volts
BB +3.3V AUX   Normal    OK   3.253 Volts
BB +1.05V P1Vccp   Normal    OK   0.695 Volts
BB +1.5 P1DDR AB   All deasserted   Unknown   Not Available
BB +1.5 P1DDR CD   All deasserted   Unknown   Not Available
BB +1.8V AUX   Normal    OK   1.801 Volts
BB +1.1V STBY   Normal    OK   1.076 Volts
BB VBAT   Normal    OK   3.017 Volts
BB +1.35 P1LV AB   Normal    OK   1.341 Volts
BB +1.35 P1LV CD   Normal    OK   1.347 Volts
HDD 0 Status   All deasserted   OK   0x0000
HDD 1 Status   All deasserted   OK   0x0000
HDD 2 Status   Drive Presence    OK   0x0001
HDD 3 Status   All deasserted   OK   0x0000
HDD 4 Status   All deasserted   OK   0x0000
HDD 5 Status   All deasserted   OK   0x0000
HDD 6 Status   All deasserted   OK   0x0000
HDD 7 Status   All deasserted   OK   0x0000
NM Capabilities   All deasserted   OK   0x0003
P1 MTT   Normal    OK   0
PSU1 Status   reports the power supply's presence has been detected    OK   0x0001
PSU2 Status   reports the power supply's presence has been detected    OK   0x0001 

[jwegman]

The BMC uses various sensors in the respective cooling domains to ramp up the fans assigned to that domain via a closed loop system.  That won't happen when things are not hooked up correctly.  There are two cooling domains, the PCI cards, and the CPUs.  The lower 120mm fan is dedicated to the PCI domain, the top to the CPU domain.

 

A W E S O M E . . . now it has gone dead silent.  48 watts on PSU1 that is unbelievable ! ! !

 

The case was open before when I didn't like the sys1 and sys2 fan noise.  This p4000 with Intel 2600cp2 combo is brilliant, but you have to get it wired and updated properly.  Thanks Jake, for the tips.

 

Now I still show an issue with system fan 2, but since I have no pci cards installed, maybe it was smart enough to shut it down.

 

Finally, the big question.  One of my other 2600 based systems isn't in a p4000, and doesn't have this quality of PSU but has dual processors and 64gb RAM, and is pulling about 200 watts at idle.  I think I want to it it in a p4000 just like this.  I like the 48 watt idle on this test system (and it is confirmed by the reading on my APC UPS with one 2670 cpu, a 7200 rpm Seagate 3TB and 32gb RAM.)

 

What is the lowest power draw you have seen with dual cpu's installed??  Can we get the 2nd cpu to go into deep sleep mode and only wake up when there is serious work to do?

 

I am using Xenserver 6.5 and Linux VM's

 

Sensor Readings below...
Pwr Unit Status   All deasserted   OK   0x0000
Pwr Unit Redund   reports full redundancy has been regained    OK   0x0001
IPMI Watchdog   All deasserted   OK   0x0000
Physical Scrty   All deasserted   OK   0x0000
FP NMI Diag Int   All deasserted   OK   0x0000
SMI Timeout   All deasserted   OK   0x0000
System Event Log   All deasserted   OK   0x0000
System Event   All deasserted   OK   0x0000
Button   All deasserted   OK   0x0000
VR Watchdog   All deasserted   OK   0x0000
SSB Therm Trip   All deasserted   OK   0x0000
BMC FW Health   All deasserted   OK   0x0000
System Airflow   Normal    OK   70 CFM
BB EDGE Temp   Normal    OK   23 degrees C
Front Panel Temp   Normal    OK   24 degrees C
SSB Temp   Normal    OK   43 degrees C
BB BMC Temp   Normal    OK   39 degrees C
BB P2 VR Temp   Normal    OK   23 degrees C
BB MEM VR Temp   Normal    OK   27 degrees C
HSBP 1 Temp   Normal    OK   24 degrees C
Exit Air Temp   Normal    OK   24 degrees C
LAN NIC Temp   Normal    OK   53 degrees C
System Fan 1   Normal    OK   1092 RPM
System Fan 2   All deasserted   Unknown   Not Available
PS1 Status   reports the power supply's presence has been detected    OK   0x0001
PS2 Status   reports the power supply's presence has been detected    OK   0x0001
PS1 Power In   Normal    OK   48 Watts
PS2 Power In   Normal    OK   4 Watts
PS1 Curr Out %   Normal    OK   4
PS2 Curr Out %   Normal    OK   0
PS1 Temperature   Normal    OK   27 degrees C
PS2 Temperature   Normal    OK   33 degrees C
HSC1 Status   All deasserted   OK   0x0000
P1 Status   reports the processor's presence has been detected    OK   0x0080
P1 Therm Margin   Normal    OK   -55 degrees C
P1 Therm Ctrl %   Normal    OK   0
P1 ERR2   All deasserted   OK   0x0000
CATERR   All deasserted   OK   0x0000
P1 MSID Mismatch   All deasserted   OK   0x0000
CPU Missing   All deasserted   OK   0x0000
P1 DTS Therm Mgn   Normal    OK   -55 degrees C
P1 VRD Hot   All deasserted   OK   0x0000
P1 MEM01 VRD Hot   All deasserted   OK   0x0000
P1 MEM23 VRD Hot   All deasserted   OK   0x0000
PS1 Fan Fail   All deasserted   OK   0x0000
PS2 Fan Fail   All deasserted   OK   0x0000
DIMM Thrm Mrgn 1   Normal    OK   -51 degrees C
DIMM Thrm Mrgn 2   Normal    OK   -51 degrees C
Mem P1 Thrm Trip   All deasserted   OK   0x0000
Agg Thrm Mgn 1   Normal    OK   -30 degrees C
Agg Thrm Mgn 2   Normal    OK   -26 degrees C
BB +12.0V   Normal    OK   11.935 Volts
BB +5.0V   Normal    OK   5.002 Volts
BB +3.3V   Normal    OK   3.382 Volts
BB +5.0V STBY   Normal    OK   4.915 Volts
BB +3.3V AUX   Normal    OK   3.253 Volts
BB +1.05V P1Vccp   Normal    OK   0.695 Volts
BB +1.5 P1DDR AB   All deasserted   Unknown   Not Available
BB +1.5 P1DDR CD   All deasserted   Unknown   Not Available
BB +1.8V AUX   Normal    OK   1.801 Volts
BB +1.1V STBY   Normal    OK   1.076 Volts
BB VBAT   Normal    OK   3.017 Volts
BB +1.35 P1LV AB   Normal    OK   1.341 Volts
BB +1.35 P1LV CD   Normal    OK   1.347 Volts
HDD 0 Status   All deasserted   OK   0x0000
HDD 1 Status   All deasserted   OK   0x0000
HDD 2 Status   Drive Presence    OK   0x0001
HDD 3 Status   All deasserted   OK   0x0000
HDD 4 Status   All deasserted   OK   0x0000
HDD 5 Status   All deasserted   OK   0x0000
HDD 6 Status   All deasserted   OK   0x0000
HDD 7 Status   All deasserted   OK   0x0000
NM Capabilities   All deasserted   OK   0x0003
P1 MTT   Normal    OK   0
PSU1 Status   reports the power supply's presence has been detected    OK   0x0001
PSU2 Status   reports the power supply's presence has been detected    OK   0x0001 

 

If you have that BIOS setting called Quite Mode (or something like that) Enabled in the Fan Settings page, I think it'll prevent some fan sensors from reporting.  I never run that Enabled, so I'm not quite sure what to expect...  I run the Performance (vs Acoustic) profile, and my PCI fan idles at ~1582rpm, and the CPU fan idles at ~1260rpm.  My typical idle airflow is ~90cfm.

 

My primary box is loaded up with drives (6x8TB seagates, 4x250GB SSD, 1x1TB SSD) and PCIe cards (2x8port SAS, 2xUSB3, 1x4port NIC, 1xGT730), and with several VMs running (currently sitting idle), I pull on average ~128watts, peak at 297watts, and minimum 86watts (per the BMC power stats for the past 116 hours).

 

 

[grimlock49]

Stupid question but how do you guys know these CPU's came from Facebook?

[BRiT]

Stupid question but how do you guys know these CPU's came from Facebook?

Facebook blogged about their recent cpu upgrades, so its conjecture thats where they cam from. What is known is they must have been from one of the major players with so many cpus available to cause the prices to fall so drastically.