Jump to content
CowboyRedBeard

SuperMicro Build - A few questions

26 posts in this topic Last Reply

Recommended Posts

So I've probably jumped in head first here, but after doing a little digging I impulsively bought a SuperMicro MB w/ CPUs combo off eBay.

 

It's a SuperMicro X9DRi-LN4F+ with dual Xeon e5-2690v2. I went this route hoping to have enough unraid to not need to upgrade until something breaks, this is definitely more server than I'll need in the near future.

 

https://www.supermicro.com/products/motherboard/Xeon/C600/X9DRi-LN4F_.cfm

https://ark.intel.com/products/75279/Intel-Xeon-Processor-E5-2690-v2-25M-Cache-3-00-GHz- 

 

So I started the build based on this guys thread on Reddit:
https://imgur.com/gallery/3EKxo

 

Which was based on this thread:

https://imgur.com/a/N48hv

 

I was actually able to do a little cleaner (IMO) install of the board and fans. They had used rubber feet to mount the top of the board where the SM form factor didn't line up with standoffs in the case. I initially tried that, but it left the board under some stress as it wasn't sitting completely flat with the difference in height between the feet and the case standoffs. Additionally, their routing of cables at the top, at least in my situation, was causing the board to be flexed further. SO... I drilled and taped holes to match the SM form factor where it was possible. Which meant I could mount the board flat and firmly in almost all areas.

k1HP7Nk.jpg

 

Which in a few cases... I mean, C'mon SuperMicro... you couldn't go with the norm here??

Bt6buii.jpg

 

There was 1 spot on the top near the IO panel that I had to use a rubber foot spacer to support the board, but it worked perfectly.

 

I also routed the power cables through a hole I created in the top bay on the chassis where I was using a fan controller that didn't take up the whole bay. This left the power cables with FAR less stress on them, and also allowed me to position exhaust fans directly over the middle of the board. I've also now got enough room to add a 3rd 120mm fan at the top of the case.

 

M8R1T4p.jpg

 

Those fans are tight, but everything fits and there's little to no stress on the cables:

Kh6etAD.jpg

 

 

Now on to issues:

 

CPU cooler issues:
I bought these thinking they'd be enough and I liked the low profile to save space. However, these processors are 130w TDP and it appears based on reading that these coolers won't be enough. Although, I've not been able to get any load on the box yet to see for sure. At idle they stay pretty cool, mid 30's. What do you guys think?

These are the coolers I got:

https://noctua.at/en/nh-l9x65/specification

 

I liked how they were low profile, but I'm not sure if they'll be large enough??

 

I think I need something like this:
https://noctua.at/en/products/cpu-cooler-workstation-server/nh-d9dx-i4-3u/specification

 

I am most concerned with making sure my temps stay super chill. So budget, within reason, isn't a concern. As you can see I bough Noctua and they aren't cheap. And you can sorta see the packaging restraints in the picture. Suggestions on alternatives?

 

 

IPMI:
I've been chatting with another member who had a SM board and I'm learning about IPMI. As you might have garnered, I'm no server or *nix expert. But it seems that SM has some strange stuff going on with fan control. I'm controlling case fans with a Thermaltake fan controller, but the CPU fans seem to just go full throttle on boot of unraid. And I'm not sure if that's because it's THESE fans... or just something else. I've installed a trial key and used that with IPMI tools in limited testing of this rig so far. I'd ideally like to take my time and get all the bugs worked out before moving my drives and existing unraid install over to this system.

I've searched, but can you guys give some advice on this IPMI situation? I want to make sure I've got this topic and the CPU coolers situated before I go to testing.

 

 

Testing:

My unraid basically spends it's life transcoding for Plex, file server and a few VMs. Transcoding can be at time taxing on my existing system as I have remote users that watch movies at 1080 or 720 and the files I have are 2160p / 4k. I plan to experiment with a GPU to do HW transcoding in the future. But... all that to say, what's a good way to stress test the system once I get this fan / IPMI / CPU cooler stuff worked out? I saw a thread about "stress" shell utility on Ubuntu, is that a good route? How do you also watch CPU clock speeds and temps to ensure you are getting the beans without over heating or thermal throttling?

 

 

What else am I missing?

Edited by CowboyRedBeard

Share this post


Link to post

OK... Now I'm in need of some help.

 

I'm not sure if this is related to the IPMI tools I installed, or if I've damaged something on the board during the install. But I don't get video on the VGA port. It does a series of boot beeps (5 beeps, followed by a couple of short ones). The system will boot and unraid comes up (I can access it via the LAN)... but no VGA.

 

I've tried clearing the CMOS to see if I've messed something up.

 

Ideas?

Share this post


Link to post

The BMC should get a separate ip from you router. Try connecting to the IPMI using network settings.  If you have enabled iGPU hardware transcoding by modprobe then the ipmi console redirection won't work anymore since the gpu is in use. I lose my ipmi console once Unraid boots. Also the vga might not work since you have to set the iGPU as the main video adapter. If all else fails you could try factory reseting BIOS and BMC.

Share this post


Link to post

I'm sorry... but I don't know what that means... I'm super n00b on this stuff, sorry!

 

So do I need to have a LAN connection for both the OS and the MB iKVM port??

I presently have only the green link plugged in.


I DID have the IPMI plugin working at one point, but after I rebooted it seemed to stop. I've not done anything apart from install that plugin and then select what I wanted to show up on the dashboard / footer.

 

 

lan.png

Share this post


Link to post

No you don't need the ipmi dedicated port plugged in if it's setup in a shared port configuration.

Share this post


Link to post

This is what shows up in the logs if it means anything.... Sorry for all the n00b questions, I'm just unfamiliar with all this.

 

Thanks for the help!

 

Feb 11 20:27:21 Tower kernel: IPMI System Interface driver.
Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol)
Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on
Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification.  The IPMI specification defines a
Feb 11 20:27:41 Tower root: # number vendors for system management.  The features of IPMI that
Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL).  The FreeIPMI
Feb 11 20:27:41 Tower root: # into FreeIPMI.
Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin 

 

Also, I saw this in the plugin support thread:

 

root@Tower:~# dmidecode -t 2
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
	Manufacturer: Supermicro
	Product Name: X9DRi-LN4+/X9DR3-LN4+
	Version: REV:1.20A
	Serial Number: 
	Asset Tag: IBM SoftLayer
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: To be filled by O.E.M.
	Chassis Handle: 0x0003
	Type: Motherboard
	Contained Object Handles: 0

 

And when I check the DHCP leases on my firewall, I don't see anything other than my unraid server under that MAC address... or actually anything with that vendor code (first 6) MAC.

Edited by CowboyRedBeard

Share this post


Link to post

Also maybe something happened when you restarted or the bmc is locked up somehow. You might try powering completely down and physically unplugging it from the wall, then holding down the power button to dissipate any residual charge. This is to make sure there's no power to the BMC at all. Then plug it back in and restart. 

Share this post


Link to post

I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper.


After, it booted, no sounds... and then powered off and came back on. Same result at the end.

Then I tried doing the BMC jumper on the board set to disable. No dice, same.

 

Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal.

Share this post


Link to post
4 minutes ago, CowboyRedBeard said:

I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper.


After, it booted, no sounds... and then powered off and came back on. Same result at the end.

Then I tried doing the BMC jumper on the board set to disable. No dice, same.

 

Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal.

So you don't get any vga at bios boot or just Unraid boot? It could be a function BIOS settings. BMC is always on. So that's why you have to unplug the power from the wall to reset it.  If you don't see sensor readings then no commands would probably work either. Also you may want to turn on shared lan interface if you are not connecting a separate ethernet cable. It should get its own ip address either way. You should be able to tweak some basic network settings in the bios for ipmi.

Share this post


Link to post

Also you should see more in the log about ipmi. This is what mine shows

Jan  6 22:08:47 Server kernel: IPMI System Interface driver.
Jan  6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
Jan  6 22:08:47 Server kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
Jan  6 22:08:47 Server kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: [io  0x0ca2] regsize 1 spacing 1 irq 0
Jan  6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI
Jan  6 22:08:47 Server kernel: ipmi_si: Adding ACPI-specified kcs state machine
Jan  6 22:08:47 Server kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
Jan  6 22:08:47 Server kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x0884, dev_id: 0x20)
Jan  6 22:08:47 Server kernel: IPMI SSIF Interface driver

It may not all be together in the log. The point I was making about getting a network connection working was so you can access the Supermicro ipmi web interface. You can access it wether you server is running or not. If your server locks up you can restart it.

Share this post


Link to post

No VGA ever, even at BIOS boot...

 

So I did a grep for lowercase 'ipmi' and get the following:
 

tower:~$ grep IPMI Desktop/towerlog 
Feb 11 20:27:21 Tower kernel: IPMI System Interface driver.
Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol)
Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on
Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification.  The IPMI specification defines a
Feb 11 20:27:41 Tower root: # number vendors for system management.  The features of IPMI that
Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL).  The FreeIPMI
Feb 11 20:27:41 Tower root: # into FreeIPMI.
Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin 
tower:~$ grep ipmi Desktop/towerlog 
Feb 11 20:27:21 Tower kernel: ipmi message handler version 39.2
Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
Feb 11 20:27:21 Tower kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
Feb 11 20:27:21 Tower kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Feb 11 20:27:21 Tower kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: Interface detection failed
Feb 11 20:27:36 Tower nerdpack: Installing ipmitool-1.8.17 package...
Feb 11 20:27:36 Tower root: Installing ipmitool-1.8.17 package... 
Feb 11 20:27:41 Tower root: plugin: installing: /boot/config/plugins/ipmi.plg
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz already exists
Feb 11 20:27:41 Tower root: plugin: running: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz
Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz
Feb 11 20:27:41 Tower root: Verifying package freeipmi-1.5.7-x86_64-2.txz.
Feb 11 20:27:41 Tower root: Installing package freeipmi-1.5.7-x86_64-2.txz:
Feb 11 20:27:41 Tower root: Executing install script for freeipmi-1.5.7-x86_64-2.txz.
Feb 11 20:27:41 Tower root: Package freeipmi-1.5.7-x86_64-2.txz installed.
Feb 11 20:27:41 Tower root: plugin: skipping: freeipmi-1.4.11-x86_64-3.txz - Unraid version too high, requires at most version 6.1.99
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi.cfg already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/fan.cfg already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz already exists
Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.md5 already exists
Feb 11 20:27:41 Tower root: plugin: creating: /tmp/start_ipmi - from INLINE content
Feb 11 20:27:41 Tower root: plugin: setting: /tmp/start_ipmi - mode to 0770
Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz
Feb 11 20:27:41 Tower root: Verifying package ipmi-2019.02.08-x86_64-1.txz.
Feb 11 20:27:41 Tower root: Installing package ipmi-2019.02.08-x86_64-1.txz:
Feb 11 20:27:41 Tower root: # The ipmi plugin allows you to view your system sensors and
Feb 11 20:27:41 Tower root: # events using your ipmi hardware. Allows for local or remote 
Feb 11 20:27:41 Tower root: Executing install script for ipmi-2019.02.08-x86_64-1.txz.
Feb 11 20:27:41 Tower root: Package ipmi-2019.02.08-x86_64-1.txz installed.
Feb 11 20:27:41 Tower root: loading ipmi drivers...
Feb 11 20:27:41 Tower kernel: ipmi device interface
Feb 11 20:27:41 Tower root: starting ipmi services...
Feb 11 20:27:42 Tower root:  ipmi has been installed.
Feb 11 20:27:45 Tower ipmifan[8201]: process started. To terminate it, type: ipmifan --quit

 

Edited by CowboyRedBeard

Share this post


Link to post

It should say found like mine did. I would try rechecking the jumpers you might have changed. Even reset the bios. If you can get into the bios, then set up the bmc network settings from there. The reset of the bmc on the board I believe only restarts it. It doesn't factory reset it. I would also try connecting another ethernet cable to the dedicated port to see if it gets an ip. I would do all that without the Unraid flash drive in.

Share this post


Link to post

I do recall seeing it having a 10.x.x.x address in BIOS... I guess I'll have to run a scan to see what that address was since I don't remember. Once I know what that address is, how do I fix this?

 

I did try to reset the CMOS... Perhaps I failed at that? The boot behavior was different after though.

Share this post


Link to post
I do recall seeing it having a 10.x.x.x address in BIOS... I guess I'll have to run a scan to see what that address was since I don't remember. Once I know what that address is, how do I fix this?
 
I did try to reset the CMOS... Perhaps I failed at that? The boot behavior was different after though.
I would also try and figure out what the beeps mean from the appendix in the motherboard manual. Usually means ram issues. After testing the bios, I would try checking with only a single stick installed or you might need 2. Try until you boot with video to the vga port. Could be a bad stick or if you messed with the cpus, could be a bent pin.

Share this post


Link to post

The manual isn't really clear on the beeps IMO

 

This is what it says:

image.png.448b1a3d4ecba563d9d21e889117b2b4.png

 

Which it does this sorta 5 beeps, followed by 2 short beeps and then another short beep of a different sound.

 

I suspect it's the "no con-out devices" ?

 

I did try it with only 1 DIMM for each CPU, and I get the same results. Also, it does actually boot unraid if I let it go. And when it's booted, it shows all the memory as there....??

 

What's the significance of the IPMI IP address? Can I browse to it or something?

Share this post


Link to post
4 hours ago, CowboyRedBeard said:

What's the significance of the IPMI IP address? Can I browse to it or something?

You should be able to browse to it with a web browser, and if it's working properly there should be a remote access option that will allow you to see what typically would be on the attached monitor.

Share this post


Link to post

OK.... so connected my laptop to the IPMI port on the server

 

Then flood ping to find the host address:

fping -a -s -r 1 -g 10.0.0.0/8

Bamn.. hit it in seconds

 

10.0.0.12

 

Tried browsing to it, nothing. Connection rejected.

 

nmap -p 1-1024 10.0.0.12

Ends up in:

 

All 1024 scanned ports on laptop (10.0.0.12) are closed

 

I'm not sure what the heck is going on here.


As stated above, I did TRY to reset the BIOS. According to the manual the steps are:

 

  1. Power down
  2. Unplug the PSU
  3. Remove CMOS battery
  4. Short the pads on the MB with a screwdriver or similar
  5. Replace CMOS battery
  6. Boot up

I did that. And when I booted back up, it powered on, turned on all the fans and then rebooted and I was back to the same thing again.

 

I have checked all those jumpers, the BCM and the VGA ones mentioned in the manual:

image.png.d7d08847bff22af9be0edf335020f5d0.png

 

image.thumb.png.571e9d5832ff08be5a22a361b5149caa.png

 

 

Now I believe I tried those with BMC disabled and VGA on. And, how they are presently.... BMC on VGA on.

 

image.thumb.png.e7e0dd98f3f61906a7b2dea32833857c.png

 

 

I'm running a fping scan presently of the entire 10.x.x.x/8 network.... JUST in case that 10.0.0.12 isn't the right address. I mean, there's nothing on that network but my laptop and the IPMI port of the server. So... 

 

But any other ideas?

Share this post


Link to post

Well... I don't know WTH 10.0.0.12 is... It's not on my machine, and it doesn't respond to anything.

 

I did find this tool on the SM website and ran it to scan the entire 10.x.x.x (I know the screenshot doesn't show that) space and it came up with nothing:image.thumb.png.b4e8b40d68f720f885a1381b0c36b2af.png
 

Share this post


Link to post

So... I plugged in a GPU just to see... and whammy... I get this:

 

image.thumb.png.11b368dc772285d7c95f7c20bd260708.png

 

image.thumb.png.0a9c952c08b5e3702bc91af7ce56a202.png

 

BIOS revision for the MB is 3.2 and 3.3 is the latest. Thinking of updating that, and perhaps the BMC/IPMI firmware as well...

Share this post


Link to post

Well... BIOS update successful ... IPMI update fail...

 

I'm going to bed...

 

image.thumb.png.f8d5c60a25baa6c5e10eb5d9e3a09fa8.png

Edited by CowboyRedBeard

Share this post


Link to post

There is usually a bmc heartbeat led. Check for that. There is a dos app to flash the ipmi (IPMICFG.exe –fd). You might look into that. There's also RKCSFlsh.exe tool. Just check the supermicro faqs for you problem.

I don't know if it's possible on the sm board but I did replace the bmc chip on an ASRock board through their support. It just dropped right in. It didn't fix the problem of missing sensors though.
It looks like yours is that nuvoton chip though. Did you try looking in the ipmi network settings in the bios?

Share this post


Link to post

The green light in the picture is the BMC heartbeat LED.... So that's "working". Will try to flash with those tools you mentioned when I get home from work tonight. The large chip in the center is the BMC, so that's beyond my skill set to replace if it doesn't work I guess I'll be looking to do a return. I'm not sure what else can be done. The IPMI doesn't initialize according to BIOS, if flashing fails then I'm guessing there's something physically wrong with it? I mean it worked for a few boot cycles after installing the unraid plugin, and I didn't make any physical changes from there out (board was already mounted in the case). 😕

 

image.thumb.png.e7e0dd98f3f61906a7b2dea32833857c.png

Share this post


Link to post

Well.... there's something wrong with it I suppose. It failed to find BMC device on the IPMICFG and the RKCS bombed out also.

 

I'm going to ask for a refund and start over I guess.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now