CowboyRedBeard Posted February 11, 2019 Share Posted February 11, 2019 So I've probably jumped in head first here, but after doing a little digging I impulsively bought a SuperMicro MB w/ CPUs combo off eBay. It's a SuperMicro X9DRi-LN4F+ with dual Xeon e5-2690v2. I went this route hoping to have enough unraid to not need to upgrade until something breaks, this is definitely more server than I'll need in the near future. https://www.supermicro.com/products/motherboard/Xeon/C600/X9DRi-LN4F_.cfm https://ark.intel.com/products/75279/Intel-Xeon-Processor-E5-2690-v2-25M-Cache-3-00-GHz- So I started the build based on this guys thread on Reddit:https://imgur.com/gallery/3EKxo Which was based on this thread: https://imgur.com/a/N48hv I was actually able to do a little cleaner (IMO) install of the board and fans. They had used rubber feet to mount the top of the board where the SM form factor didn't line up with standoffs in the case. I initially tried that, but it left the board under some stress as it wasn't sitting completely flat with the difference in height between the feet and the case standoffs. Additionally, their routing of cables at the top, at least in my situation, was causing the board to be flexed further. SO... I drilled and taped holes to match the SM form factor where it was possible. Which meant I could mount the board flat and firmly in almost all areas. Which in a few cases... I mean, C'mon SuperMicro... you couldn't go with the norm here?? There was 1 spot on the top near the IO panel that I had to use a rubber foot spacer to support the board, but it worked perfectly. I also routed the power cables through a hole I created in the top bay on the chassis where I was using a fan controller that didn't take up the whole bay. This left the power cables with FAR less stress on them, and also allowed me to position exhaust fans directly over the middle of the board. I've also now got enough room to add a 3rd 120mm fan at the top of the case. Those fans are tight, but everything fits and there's little to no stress on the cables: Now on to issues: CPU cooler issues: I bought these thinking they'd be enough and I liked the low profile to save space. However, these processors are 130w TDP and it appears based on reading that these coolers won't be enough. Although, I've not been able to get any load on the box yet to see for sure. At idle they stay pretty cool, mid 30's. What do you guys think? These are the coolers I got: https://noctua.at/en/nh-l9x65/specification I liked how they were low profile, but I'm not sure if they'll be large enough?? I think I need something like this:https://noctua.at/en/products/cpu-cooler-workstation-server/nh-d9dx-i4-3u/specification I am most concerned with making sure my temps stay super chill. So budget, within reason, isn't a concern. As you can see I bough Noctua and they aren't cheap. And you can sorta see the packaging restraints in the picture. Suggestions on alternatives? IPMI: I've been chatting with another member who had a SM board and I'm learning about IPMI. As you might have garnered, I'm no server or *nix expert. But it seems that SM has some strange stuff going on with fan control. I'm controlling case fans with a Thermaltake fan controller, but the CPU fans seem to just go full throttle on boot of unraid. And I'm not sure if that's because it's THESE fans... or just something else. I've installed a trial key and used that with IPMI tools in limited testing of this rig so far. I'd ideally like to take my time and get all the bugs worked out before moving my drives and existing unraid install over to this system. I've searched, but can you guys give some advice on this IPMI situation? I want to make sure I've got this topic and the CPU coolers situated before I go to testing. Testing: My unraid basically spends it's life transcoding for Plex, file server and a few VMs. Transcoding can be at time taxing on my existing system as I have remote users that watch movies at 1080 or 720 and the files I have are 2160p / 4k. I plan to experiment with a GPU to do HW transcoding in the future. But... all that to say, what's a good way to stress test the system once I get this fan / IPMI / CPU cooler stuff worked out? I saw a thread about "stress" shell utility on Ubuntu, is that a good route? How do you also watch CPU clock speeds and temps to ensure you are getting the beans without over heating or thermal throttling? What else am I missing? Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 OK... Now I'm in need of some help. I'm not sure if this is related to the IPMI tools I installed, or if I've damaged something on the board during the install. But I don't get video on the VGA port. It does a series of boot beeps (5 beeps, followed by a couple of short ones). The system will boot and unraid comes up (I can access it via the LAN)... but no VGA. I've tried clearing the CMOS to see if I've messed something up. Ideas? Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 Also, when I boot unraid the IPMI tools doesn't show any of the sensors any longer? Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 The BMC should get a separate ip from you router. Try connecting to the IPMI using network settings. If you have enabled iGPU hardware transcoding by modprobe then the ipmi console redirection won't work anymore since the gpu is in use. I lose my ipmi console once Unraid boots. Also the vga might not work since you have to set the iGPU as the main video adapter. If all else fails you could try factory reseting BIOS and BMC. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 I'm sorry... but I don't know what that means... I'm super n00b on this stuff, sorry! So do I need to have a LAN connection for both the OS and the MB iKVM port?? I presently have only the green link plugged in. I DID have the IPMI plugin working at one point, but after I rebooted it seemed to stop. I've not done anything apart from install that plugin and then select what I wanted to show up on the dashboard / footer. Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 No you don't need the ipmi dedicated port plugged in if it's setup in a shared port configuration. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 This is what shows up in the logs if it means anything.... Sorry for all the n00b questions, I'm just unfamiliar with all this. Thanks for the help! Feb 11 20:27:21 Tower kernel: IPMI System Interface driver. Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol) Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification. The IPMI specification defines a Feb 11 20:27:41 Tower root: # number vendors for system management. The features of IPMI that Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL). The FreeIPMI Feb 11 20:27:41 Tower root: # into FreeIPMI. Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin Also, I saw this in the plugin support thread: root@Tower:~# dmidecode -t 2 # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: Supermicro Product Name: X9DRi-LN4+/X9DR3-LN4+ Version: REV:1.20A Serial Number: Asset Tag: IBM SoftLayer Features: Board is a hosting board Board is replaceable Location In Chassis: To be filled by O.E.M. Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0 And when I check the DHCP leases on my firewall, I don't see anything other than my unraid server under that MAC address... or actually anything with that vendor code (first 6) MAC. Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 Also maybe something happened when you restarted or the bmc is locked up somehow. You might try powering completely down and physically unplugging it from the wall, then holding down the power button to dissipate any residual charge. This is to make sure there's no power to the BMC at all. Then plug it back in and restart. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper. After, it booted, no sounds... and then powered off and came back on. Same result at the end. Then I tried doing the BMC jumper on the board set to disable. No dice, same. Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal. Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 4 minutes ago, CowboyRedBeard said: I did try clearing the CMOS per the manual, power down, unplug, remove battery, short the jumper. After, it booted, no sounds... and then powered off and came back on. Same result at the end. Then I tried doing the BMC jumper on the board set to disable. No dice, same. Are there some shell commands that will give me info on the IPMI status or something I can do to debug? Because the system seems to work just fine apart from the VGA not showing a signal. So you don't get any vga at bios boot or just Unraid boot? It could be a function BIOS settings. BMC is always on. So that's why you have to unplug the power from the wall to reset it. If you don't see sensor readings then no commands would probably work either. Also you may want to turn on shared lan interface if you are not connecting a separate ethernet cable. It should get its own ip address either way. You should be able to tweak some basic network settings in the bios for ipmi. Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 Also you should see more in the log about ipmi. This is what mine shows Jan 6 22:08:47 Server kernel: IPMI System Interface driver. Jan 6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS Jan 6 22:08:47 Server kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 Jan 6 22:08:47 Server kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Jan 6 22:08:47 Server kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI Jan 6 22:08:47 Server kernel: ipmi_si IPI0001:00: [io 0x0ca2] regsize 1 spacing 1 irq 0 Jan 6 22:08:47 Server kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI Jan 6 22:08:47 Server kernel: ipmi_si: Adding ACPI-specified kcs state machine Jan 6 22:08:47 Server kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 Jan 6 22:08:47 Server kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x0884, dev_id: 0x20) Jan 6 22:08:47 Server kernel: IPMI SSIF Interface driver It may not all be together in the log. The point I was making about getting a network connection working was so you can access the Supermicro ipmi web interface. You can access it wether you server is running or not. If your server locks up you can restart it. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 No VGA ever, even at BIOS boot... So I did a grep for lowercase 'ipmi' and get the following: tower:~$ grep IPMI Desktop/towerlog Feb 11 20:27:21 Tower kernel: IPMI System Interface driver. Feb 11 20:27:41 Tower root: # FreeIPMI (GNU implementation of the IPMI protocol) Feb 11 20:27:41 Tower root: # FreeIPMI provides in-band and out-of-band IPMI software based on Feb 11 20:27:41 Tower root: # the IPMI v1.5/2.0 specification. The IPMI specification defines a Feb 11 20:27:41 Tower root: # number vendors for system management. The features of IPMI that Feb 11 20:27:41 Tower root: # monitoring, power control, and serial-over-LAN (SOL). The FreeIPMI Feb 11 20:27:41 Tower root: # into FreeIPMI. Feb 11 20:27:41 Tower root: # IPMI unRAID Plugin tower:~$ grep ipmi Desktop/towerlog Feb 11 20:27:21 Tower kernel: ipmi message handler version 39.2 Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS Feb 11 20:27:21 Tower kernel: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 Feb 11 20:27:21 Tower kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Feb 11 20:27:21 Tower kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 Feb 11 20:27:21 Tower kernel: ipmi_si dmi-ipmi-si.0: Interface detection failed Feb 11 20:27:36 Tower nerdpack: Installing ipmitool-1.8.17 package... Feb 11 20:27:36 Tower root: Installing ipmitool-1.8.17 package... Feb 11 20:27:41 Tower root: plugin: installing: /boot/config/plugins/ipmi.plg Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz already exists Feb 11 20:27:41 Tower root: plugin: running: /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/freeipmi-1.5.7-x86_64-2.txz Feb 11 20:27:41 Tower root: Verifying package freeipmi-1.5.7-x86_64-2.txz. Feb 11 20:27:41 Tower root: Installing package freeipmi-1.5.7-x86_64-2.txz: Feb 11 20:27:41 Tower root: Executing install script for freeipmi-1.5.7-x86_64-2.txz. Feb 11 20:27:41 Tower root: Package freeipmi-1.5.7-x86_64-2.txz installed. Feb 11 20:27:41 Tower root: plugin: skipping: freeipmi-1.4.11-x86_64-3.txz - Unraid version too high, requires at most version 6.1.99 Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi.cfg already exists Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/fan.cfg already exists Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz already exists Feb 11 20:27:41 Tower root: plugin: skipping: /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.md5 already exists Feb 11 20:27:41 Tower root: plugin: creating: /tmp/start_ipmi - from INLINE content Feb 11 20:27:41 Tower root: plugin: setting: /tmp/start_ipmi - mode to 0770 Feb 11 20:27:41 Tower root: | Installing new package /boot/config/plugins/ipmi/ipmi-2019.02.08-x86_64-1.txz Feb 11 20:27:41 Tower root: Verifying package ipmi-2019.02.08-x86_64-1.txz. Feb 11 20:27:41 Tower root: Installing package ipmi-2019.02.08-x86_64-1.txz: Feb 11 20:27:41 Tower root: # The ipmi plugin allows you to view your system sensors and Feb 11 20:27:41 Tower root: # events using your ipmi hardware. Allows for local or remote Feb 11 20:27:41 Tower root: Executing install script for ipmi-2019.02.08-x86_64-1.txz. Feb 11 20:27:41 Tower root: Package ipmi-2019.02.08-x86_64-1.txz installed. Feb 11 20:27:41 Tower root: loading ipmi drivers... Feb 11 20:27:41 Tower kernel: ipmi device interface Feb 11 20:27:41 Tower root: starting ipmi services... Feb 11 20:27:42 Tower root: ipmi has been installed. Feb 11 20:27:45 Tower ipmifan[8201]: process started. To terminate it, type: ipmifan --quit Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 It should say found like mine did. I would try rechecking the jumpers you might have changed. Even reset the bios. If you can get into the bios, then set up the bmc network settings from there. The reset of the bmc on the board I believe only restarts it. It doesn't factory reset it. I would also try connecting another ethernet cable to the dedicated port to see if it gets an ip. I would do all that without the Unraid flash drive in. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 I do recall seeing it having a 10.x.x.x address in BIOS... I guess I'll have to run a scan to see what that address was since I don't remember. Once I know what that address is, how do I fix this? I did try to reset the CMOS... Perhaps I failed at that? The boot behavior was different after though. Link to comment
dmacias Posted February 12, 2019 Share Posted February 12, 2019 I do recall seeing it having a 10.x.x.x address in BIOS... I guess I'll have to run a scan to see what that address was since I don't remember. Once I know what that address is, how do I fix this? I did try to reset the CMOS... Perhaps I failed at that? The boot behavior was different after though.I would also try and figure out what the beeps mean from the appendix in the motherboard manual. Usually means ram issues. After testing the bios, I would try checking with only a single stick installed or you might need 2. Try until you boot with video to the vga port. Could be a bad stick or if you messed with the cpus, could be a bent pin. Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 The manual isn't really clear on the beeps IMO This is what it says: Which it does this sorta 5 beeps, followed by 2 short beeps and then another short beep of a different sound. I suspect it's the "no con-out devices" ? I did try it with only 1 DIMM for each CPU, and I get the same results. Also, it does actually boot unraid if I let it go. And when it's booted, it shows all the memory as there....?? What's the significance of the IPMI IP address? Can I browse to it or something? Link to comment
CowboyRedBeard Posted February 12, 2019 Author Share Posted February 12, 2019 My next thought was to plug a GPU into it and see if I get a BIOS screen from that... 🤔 Link to comment
JonathanM Posted February 13, 2019 Share Posted February 13, 2019 4 hours ago, CowboyRedBeard said: What's the significance of the IPMI IP address? Can I browse to it or something? You should be able to browse to it with a web browser, and if it's working properly there should be a remote access option that will allow you to see what typically would be on the attached monitor. Link to comment
CowboyRedBeard Posted February 13, 2019 Author Share Posted February 13, 2019 OK.... so connected my laptop to the IPMI port on the server Then flood ping to find the host address: fping -a -s -r 1 -g 10.0.0.0/8 Bamn.. hit it in seconds 10.0.0.12 Tried browsing to it, nothing. Connection rejected. nmap -p 1-1024 10.0.0.12 Ends up in: All 1024 scanned ports on laptop (10.0.0.12) are closed I'm not sure what the heck is going on here. As stated above, I did TRY to reset the BIOS. According to the manual the steps are: Power down Unplug the PSU Remove CMOS battery Short the pads on the MB with a screwdriver or similar Replace CMOS battery Boot up I did that. And when I booted back up, it powered on, turned on all the fans and then rebooted and I was back to the same thing again. I have checked all those jumpers, the BCM and the VGA ones mentioned in the manual: Now I believe I tried those with BMC disabled and VGA on. And, how they are presently.... BMC on VGA on. I'm running a fping scan presently of the entire 10.x.x.x/8 network.... JUST in case that 10.0.0.12 isn't the right address. I mean, there's nothing on that network but my laptop and the IPMI port of the server. So... But any other ideas? Link to comment
CowboyRedBeard Posted February 13, 2019 Author Share Posted February 13, 2019 Well... I don't know WTH 10.0.0.12 is... It's not on my machine, and it doesn't respond to anything. I did find this tool on the SM website and ran it to scan the entire 10.x.x.x (I know the screenshot doesn't show that) space and it came up with nothing: Link to comment
CowboyRedBeard Posted February 13, 2019 Author Share Posted February 13, 2019 So... I plugged in a GPU just to see... and whammy... I get this: BIOS revision for the MB is 3.2 and 3.3 is the latest. Thinking of updating that, and perhaps the BMC/IPMI firmware as well... Link to comment
CowboyRedBeard Posted February 13, 2019 Author Share Posted February 13, 2019 Well... BIOS update successful ... IPMI update fail... I'm going to bed... Link to comment
dmacias Posted February 13, 2019 Share Posted February 13, 2019 There is usually a bmc heartbeat led. Check for that. There is a dos app to flash the ipmi (IPMICFG.exe –fd). You might look into that. There's also RKCSFlsh.exe tool. Just check the supermicro faqs for you problem.I don't know if it's possible on the sm board but I did replace the bmc chip on an ASRock board through their support. It just dropped right in. It didn't fix the problem of missing sensors though.It looks like yours is that nuvoton chip though. Did you try looking in the ipmi network settings in the bios? Link to comment
CowboyRedBeard Posted February 13, 2019 Author Share Posted February 13, 2019 The green light in the picture is the BMC heartbeat LED.... So that's "working". Will try to flash with those tools you mentioned when I get home from work tonight. The large chip in the center is the BMC, so that's beyond my skill set to replace if it doesn't work I guess I'll be looking to do a return. I'm not sure what else can be done. The IPMI doesn't initialize according to BIOS, if flashing fails then I'm guessing there's something physically wrong with it? I mean it worked for a few boot cycles after installing the unraid plugin, and I didn't make any physical changes from there out (board was already mounted in the case). 😕 Link to comment
CowboyRedBeard Posted February 14, 2019 Author Share Posted February 14, 2019 Well.... there's something wrong with it I suppose. It failed to find BMC device on the IPMICFG and the RKCS bombed out also. I'm going to ask for a refund and start over I guess. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.