SuperMicro Build - A few questions


CowboyRedBeard

Recommended Posts

So I've probably jumped in head first here, but after doing a little digging I impulsively bought a SuperMicro MB w/ CPUs combo off eBay.

 

It's a SuperMicro X9DRi-LN4F+ with dual Xeon e5-2690v2. I went this route hoping to have enough unraid to not need to upgrade until something breaks, this is definitely more server than I'll need in the near future.

 

https://www.supermicro.com/products/motherboard/Xeon/C600/X9DRi-LN4F_.cfm

https://ark.intel.com/products/75279/Intel-Xeon-Processor-E5-2690-v2-25M-Cache-3-00-GHz- 

 

So I started the build based on this guys thread on Reddit:
https://imgur.com/gallery/3EKxo

 

Which was based on this thread:

https://imgur.com/a/N48hv

 

I was actually able to do a little cleaner (IMO) install of the board and fans. They had used rubber feet to mount the top of the board where the SM form factor didn't line up with standoffs in the case. I initially tried that, but it left the board under some stress as it wasn't sitting completely flat with the difference in height between the feet and the case standoffs. Additionally, their routing of cables at the top, at least in my situation, was causing the board to be flexed further. SO... I drilled and taped holes to match the SM form factor where it was possible. Which meant I could mount the board flat and firmly in almost all areas.

k1HP7Nk.jpg

 

Which in a few cases... I mean, C'mon SuperMicro... you couldn't go with the norm here??

Bt6buii.jpg

 

There was 1 spot on the top near the IO panel that I had to use a rubber foot spacer to support the board, but it worked perfectly.

 

I also routed the power cables through a hole I created in the top bay on the chassis where I was using a fan controller that didn't take up the whole bay. This left the power cables with FAR less stress on them, and also allowed me to position exhaust fans directly over the middle of the board. I've also now got enough room to add a 3rd 120mm fan at the top of the case.

 

M8R1T4p.jpg

 

Those fans are tight, but everything fits and there's little to no stress on the cables:

Kh6etAD.jpg

 

 

Now on to issues:

 

CPU cooler issues:
I bought these thinking they'd be enough and I liked the low profile to save space. However, these processors are 130w TDP and it appears based on reading that these coolers won't be enough. Although, I've not been able to get any load on the box yet to see for sure. At idle they stay pretty cool, mid 30's. What do you guys think?

These are the coolers I got:

https://noctua.at/en/nh-l9x65/specification

 

I liked how they were low profile, but I'm not sure if they'll be large enough??

 

I think I need something like this:
https://noctua.at/en/products/cpu-cooler-workstation-server/nh-d9dx-i4-3u/specification

 

I am most concerned with making sure my temps stay super chill. So budget, within reason, isn't a concern. As you can see I bough Noctua and they aren't cheap. And you can sorta see the packaging restraints in the picture. Suggestions on alternatives?

 

 

IPMI:
I've been chatting with another member who had a SM board and I'm learning about IPMI. As you might have garnered, I'm no server or *nix expert. But it seems that SM has some strange stuff going on with fan control. I'm controlling case fans with a Thermaltake fan controller, but the CPU fans seem to just go full throttle on boot of unraid. And I'm not sure if that's because it's THESE fans... or just something else. I've installed a trial key and used that with IPMI tools in limited testing of this rig so far. I'd ideally like to take my time and get all the bugs worked out before moving my drives and existing unraid install over to this system.

I've searched, but can you guys give some advice on this IPMI situation? I want to make sure I've got this topic and the CPU coolers situated before I go to testing.

 

 

Testing:

My unraid basically spends it's life transcoding for Plex, file server and a few VMs. Transcoding can be at time taxing on my existing system as I have remote users that watch movies at 1080 or 720 and the files I have are 2160p / 4k. I plan to experiment with a GPU to do HW transcoding in the future. But... all that to say, what's a good way to stress test the system once I get this fan / IPMI / CPU cooler stuff worked out? I saw a thread about "stress" shell utility on Ubuntu, is that a good route? How do you also watch CPU clock speeds and temps to ensure you are getting the beans without over heating or thermal throttling?

 

 

What else am I missing?

Edited by CowboyRedBeard
Link to comment

OK... Now I'm in need of some help.

 

I'm not sure if this is related to the IPMI tools I installed, or if I've damaged something on the board during the install. But I don't get video on the VGA port. It does a series of boot beeps (5 beeps, followed by a couple of short ones). The system will boot and unraid comes up (I can access it via the LAN)... but no VGA.

 

I've tried clearing the CMOS to see if I've messed something up.

 

Ideas?

  • Like 1
Link to comment

The BMC should get a separate ip from you router. Try connecting to the IPMI using network settings.  If you have enabled iGPU hardware transcoding by modprobe then the ipmi console redirection won't work anymore since the gpu is in use. I lose my ipmi console once Unraid boots. Also the vga might not work since you have to set the iGPU as the main video adapter. If all else fails you could try factory reseting BIOS and BMC.

Link to comment

I'm sorry... but I don't know what that means... I'm super n00b on this stuff, sorry!

 

So do I need to have a LAN connection for both the OS and the MB iKVM port??

I presently have only the green link plugged in.


I DID have the IPMI plugin working at one point, but after I rebooted it seemed to stop. I've not done anything apart from install that plugin and then select what I wanted to show up on the dashboard / footer.

 

 

lan.png

Link to comment

This is what shows up in the logs if it means anything.... Sorry for all the n00b questions, I'm just unfamiliar with all this.

 

Thanks for the help!

 

Feb 11 20:27:21 Tower kernel: IPMI System Interface driver.
Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol)
Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on
Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification.  The IPMI specification defines a
Feb 11 20:27:41 Tower root: # number vendors for system management.  The features of IPMI that
Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL).  The FreeIPMI
Feb 11 20:27:41 Tower root: # into FreeIPMI.
Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin 

 

Also, I saw this in the plugin support thread:

 

root@Tower:~# dmidecode -t 2
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: Supermicro
	Product Name: X9DRi-LN4+/X9DR3-LN4+
	Version: REV:1.20A
	Serial Number: 
	Asset Tag: IBM SoftLayer
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: To be filled by O.E.M.
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

 

And when I check the DHCP leases on my firewall, I don't see anything other than my unraid server under that MAC address... or actually anything with that vendor code (first 6) MAC.

Edited by CowboyRedBeard
Link to comment

Also maybe something happened when you restarted or the bmc is locked up somehow. You might try powering completely down and physically unplugging it from the wall, then holding down the power button to dissipate any residual charge. This is to make sure there's no power to the BMC at all. Then plug it back in and restart. 

Link to comment

I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper.


After, it booted, no sounds... and then powered off and came back on. Same result at the end.

Then I tried doing the BMC jumper on the board set to disable. No dice, same.

 

Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal.

Link to comment
4 minutes ago, CowboyRedBeard said:

I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper.


After, it booted, no sounds... and then powered off and came back on. Same result at the end.

Then I tried doing the BMC jumper on the board set to disable. No dice, same.

 

Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal.

So you don't get any vga at bios boot or just Unraid boot? It could be a function BIOS settings. BMC is always on. So that's why you have to unplug the power from the wall to reset it.  If you don't see sensor readings then no commands would probably work either. Also you may want to turn on shared lan interface if you are not connecting a separate ethernet cable. It should get its own ip address either way. You should be able to tweak some basic network settings in the bios for ipmi.

Link to comment

Also you should see more in the log about ipmi. This is what mine shows

Jan  6 22:08:47 Server kernel: IPMI System Interface driver.
Jan  6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
Jan  6 22:08:47 Server kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
Jan  6 22:08:47 Server kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: [io  0x0ca2] regsize 1 spacing 1 irq 0
Jan  6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI
Jan  6 22:08:47 Server kernel: ipmi_si: Adding ACPI-specified kcs state machine
Jan  6 22:08:47 Server kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x0884, dev_id: 0x20)
Jan  6 22:08:47 Server kernel: IPMI SSIF Interface driver

It may not all be together in the log. The point I was making about getting a network connection working was so you can access the Supermicro ipmi web interface. You can access it wether you server is running or not. If your server locks up you can restart it.

Link to comment

No VGA ever, even at BIOS boot...

 

So I did a grep for lowercase 'ipmi' and get the following:
 

tower:~$ grep IPMI Desktop/towerlog 
Feb 11 20:27:21 Tower kernel: IPMI System Interface driver.
Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol)
Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on
Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification.  The IPMI specification defines a
Feb 11 20:27:41 Tower root: # number vendors for system management.  The features of IPMI that
Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL).  The FreeIPMI
Feb 11 20:27:41 Tower root: # into FreeIPMI.
Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin 
tower:~$ grep ipmi Desktop/towerlog 
Feb 11 20:27:21 Tower kernel: ipmi message handler version 39.2
Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
Feb 11 20:27:21 Tower kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
Feb 11 20:27:21 Tower kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Feb 11 20:27:21 Tower kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: Interface detection failed
Feb 11 20:27:36 Tower nerdpack: Installing ipmitool-1.8.17 package...
Feb 11 20:27:36 Tower root: Installing ipmitool-1.8.17 package... 
Feb 11 20:27:41 Tower root: plugin: installing: /boot/config/plugins/ipmi.plg
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz already exists
Feb 11 20:27:41 Tower root: plugin: running: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz
Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz
Feb 11 20:27:41 Tower root: Verifying package freeipmi-1.5.7-x86_64-2.txz.
Feb 11 20:27:41 Tower root: Installing package freeipmi-1.5.7-x86_64-2.txz:
Feb 11 20:27:41 Tower root: Executing install script for freeipmi-1.5.7-x86_64-2.txz.
Feb 11 20:27:41 Tower root: Package freeipmi-1.5.7-x86_64-2.txz installed.
Feb 11 20:27:41 Tower root: plugin: skipping: freeipmi-1.4.11-x86_64-3.txz - Unraid version too high, requires at most version 6.1.99
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi.cfg already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/fan.cfg already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.md5 already exists
Feb 11 20:27:41 Tower root: plugin: creating: /tmp/start_ipmi - from INLINE content
Feb 11 20:27:41 Tower root: plugin: setting: /tmp/start_ipmi - mode to 0770
Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz
Feb 11 20:27:41 Tower root: Verifying package ipmi-2019.02.08-x86_64-1.txz.
Feb 11 20:27:41 Tower root: Installing package ipmi-2019.02.08-x86_64-1.txz:
Feb 11 20:27:41 Tower root: # The ipmi plugin allows you to view your system sensors and
Feb 11 20:27:41 Tower root: # events using your ipmi hardware. Allows for local or remote 
Feb 11 20:27:41 Tower root: Executing install script for ipmi-2019.02.08-x86_64-1.txz.
Feb 11 20:27:41 Tower root: Package ipmi-2019.02.08-x86_64-1.txz installed.
Feb 11 20:27:41 Tower root: loading ipmi drivers...
Feb 11 20:27:41 Tower kernel: ipmi device interface
Feb 11 20:27:41 Tower root: starting ipmi services...
Feb 11 20:27:42 Tower root:  ipmi has been installed.
Feb 11 20:27:45 Tower ipmifan[8201]: process started. To terminate it, type: ipmifan --quit

 

Edited by CowboyRedBeard
Link to comment

It should say found like mine did. I would try rechecking the jumpers you might have changed. Even reset the bios. If you can get into the bios, then set up the bmc network settings from there. The reset of the bmc on the board I believe only restarts it. It doesn't factory reset it. I would also try connecting another ethernet cable to the dedicated port to see if it gets an ip. I would do all that without the Unraid flash drive in.

Link to comment
I do recall seeing it having a 10.x.x.x address in BIOS... I guess I'll have to run a scan to see what that address was since I don't remember. Once I know what that address is, how do I fix this?
 
I did try to reset the CMOS... Perhaps I failed at that? The boot behavior was different after though.
I would also try and figure out what the beeps mean from the appendix in the motherboard manual. Usually means ram issues. After testing the bios, I would try checking with only a single stick installed or you might need 2. Try until you boot with video to the vga port. Could be a bad stick or if you messed with the cpus, could be a bent pin.
Link to comment

The manual isn't really clear on the beeps IMO

 

This is what it says:

image.png.448b1a3d4ecba563d9d21e889117b2b4.png

 

Which it does this sorta 5 beeps, followed by 2 short beeps and then another short beep of a different sound.

 

I suspect it's the "no con-out devices" ?

 

I did try it with only 1 DIMM for each CPU, and I get the same results. Also, it does actually boot unraid if I let it go. And when it's booted, it shows all the memory as there....??

 

What's the significance of the IPMI IP address? Can I browse to it or something?

Link to comment
4 hours ago, CowboyRedBeard said:

What's the significance of the IPMI IP address? Can I browse to it or something?

You should be able to browse to it with a web browser, and if it's working properly there should be a remote access option that will allow you to see what typically would be on the attached monitor.

Link to comment

OK.... so connected my laptop to the IPMI port on the server

 

Then flood ping to find the host address:

fping -a -s -r 1 -g 10.0.0.0/8

Bamn.. hit it in seconds

 

10.0.0.12

 

Tried browsing to it, nothing. Connection rejected.

 

nmap -p 1-1024 10.0.0.12

Ends up in:

 

All 1024 scanned ports on laptop (10.0.0.12) are closed

 

I'm not sure what the heck is going on here.


As stated above, I did TRY to reset the BIOS. According to the manual the steps are:

 

  1. Power down
  2. Unplug the PSU
  3. Remove CMOS battery
  4. Short the pads on the MB with a screwdriver or similar
  5. Replace CMOS battery
  6. Boot up

I did that. And when I booted back up, it powered on, turned on all the fans and then rebooted and I was back to the same thing again.

 

I have checked all those jumpers, the BCM and the VGA ones mentioned in the manual:

image.png.d7d08847bff22af9be0edf335020f5d0.png

 

image.thumb.png.571e9d5832ff08be5a22a361b5149caa.png

 

 

Now I believe I tried those with BMC disabled and VGA on. And, how they are presently.... BMC on VGA on.

 

image.thumb.png.e7e0dd98f3f61906a7b2dea32833857c.png

 

 

I'm running a fping scan presently of the entire 10.x.x.x/8 network.... JUST in case that 10.0.0.12 isn't the right address. I mean, there's nothing on that network but my laptop and the IPMI port of the server. So... 

 

But any other ideas?

Link to comment

There is usually a bmc heartbeat led. Check for that. There is a dos app to flash the ipmi (IPMICFG.exe –fd). You might look into that. There's also RKCSFlsh.exe tool. Just check the supermicro faqs for you problem.

I don't know if it's possible on the sm board but I did replace the bmc chip on an ASRock board through their support. It just dropped right in. It didn't fix the problem of missing sensors though.
It looks like yours is that nuvoton chip though. Did you try looking in the ipmi network settings in the bios?

Link to comment

The green light in the picture is the BMC heartbeat LED.... So that's "working". Will try to flash with those tools you mentioned when I get home from work tonight. The large chip in the center is the BMC, so that's beyond my skill set to replace if it doesn't work I guess I'll be looking to do a return. I'm not sure what else can be done. The IPMI doesn't initialize according to BIOS, if flashing fails then I'm guessing there's something physically wrong with it? I mean it worked for a few boot cycles after installing the unraid plugin, and I didn't make any physical changes from there out (board was already mounted in the case). 😕

 

image.thumb.png.e7e0dd98f3f61906a7b2dea32833857c.png

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.