[Plugin] IPMI for unRAID 6.1+


Recommended Posts

I think this plugin just solved a lot of problems for me in one go, so I think I've just bought you a beer (if a beer is more there, let me know!).

 

If I can just run through what I've done to make sure I'm not being very stupid, as this is a bit new to me...

 

My Unraid system is built on a SuperMicro X10SL7-F motherboard inside a SuperMicro SC846 chassis. Up until last week it was running stock fans and quiet PSU, but after a year of working from home, I decided it was time to make it quieter.  I replaced all the fans with...

  • 3 x Noctua 120mm 3000RPM PWM industrial fans in the fan wall.  All plugged into FAN A using a 3 way PWM molex-powered splitter
  • 2 x Noctua 80mm PWM exhaust fans in the rear, plugged into FAN 4 with 2 way PWM splitter
  • 2 x Noctua 92mm PWM fans on a Noctua CPU cooler plugged into FAN 1 with 2 way PWM splitter

As you probably know, this then resulted all fans cycling between low and full every 10 seconds due to the bios-set fan thresholds being crossed.  After researching, I arrived here and was able to tweak the lower and upper thresholds to what I think is OK using the config editor (non-critical, critical, non-recoverable)...

  • FAN 1 (2 x 92mm CPU) - Lower: 0, 100, 300. Upper: 2200, 2300, 2400
  • FAN 4 (2 x 80mm exhaust) - Lower: 0, 100, 300. Upper: 2400, 2500, 2600
  • FAN A (3 x 120mm fan wall) - Lower: 400, 500, 600. Upper: 3200, 3300, 3400

These are broadly set using the Noctua website specs, and stopped the constant cycling.  Haven't yet found a guide on what these values should be.  They were still running fast though, so I enabled fan control, which is now whisper quiet and cooler than before.  Hoping it's OK...

  • Left polling times as default
  • Unchecked my two SSDs in the "hard drives to poll" option
  • I've set FAN1234 to track the CPU temperature with temp/speed ranges left.  This bought down speeds massively, but not sure it's the right thing to do
  • Set FANA (the fan wall pulling air across the hot swap bays) to track HDD Temperature and set the temp limits to 55 high, 20 low

image.png.ee1c56dcc068176ff3db8d99f1b63f74.png

 

Just spun up all hard drives.  The CPU is at 35C, system 33C, HDD 35C.  Fans 1, 4, A are spinning at 400, 500 1200RPM.  I'm honestly over the moon about how my house is now back to normal noise, and I haven't had any temperature warnings yet.  Just looking for reassurance/guidance.

 

Thanks again!

  • Like 1
Link to comment

I'm now using HP ProLiant MicroServer Gen8, and successfully set my IPMI(using iLO, the IPMI has a different IP from the unRAID's network).

When I try 

ipmi-sensors

it says:

ID | Name             | Type         | Reading    | Units | Event
18 | UID Light        | OEM Reserved | N/A        | N/A   | 'OEM Event = 0001h'
19 | Health LED       | OEM Reserved | N/A        | N/A   | N/A
20 | 01-Inlet Ambient | Temperature  | 17.00      | C     | 'OK'
21 | 02-CPU           | Temperature  | 40.00      | C     | 'OK'
22 | 03-P1 DIMM 1-2   | Temperature  | 34.00      | C     | 'OK'
23 | 04-HD Max        | Temperature  | N/A        | C     | N/A
24 | 05-Chipset       | Temperature  | 47.00      | C     | 'OK'
25 | 06-Chipset Zone  | Temperature  | 37.00      | C     | 'OK'
26 | 07-VR P1 Zone    | Temperature  | 50.00      | C     | 'OK'
27 | 08-Supercap Max  | Temperature  | N/A        | C     | N/A
28 | 09-iLO Zone      | Temperature  | 41.00      | C     | 'OK'
29 | 10-PCI 1         | Temperature  | N/A        | C     | N/A
30 | 11-PCI 1 Zone    | Temperature  | 33.00      | C     | 'OK'
31 | 12-Sys Exhaust   | Temperature  | 39.00      | C     | 'OK'
32 | 13-LOM           | Temperature  | N/A        | C     | N/A
33 | Fan 1            | Fan          | N/A        | N/A   | 'transition to Running'
34 | Power Supply 1   | Power Supply | N/A        | N/A   | 'Presence detected'
35 | Memory           | Memory       | N/A        | N/A   | 'Presence detected'

Seems fine, but in the Reading section of the plugin, it only has

127.0.0.1	99	HDD Temperature	N/A	N/A	N/A	35°C	Temperature	45.00	50.00	N/A

Seems directly read from the unRAID itself.

 

What happened? Is it possible to import the sensors data got from the 'ipmi-sensors' command?

Edited by S4kura0ne
  • Like 1
Link to comment
  • 4 weeks later...

The IPMI plugin was working without any issues until some couple of days ago (maybe until updating unraid to latest stable version but I can't confirm this).

The problem is that when the server starts all fans running on full speed and do not change the speed. If I turn off the fan control and turn it on afterwards the fans are running normal for about...~1h and than turning back to full power. I didn't changed anything in bios or any other system related setting. This also happens when the server runs for days.
The threshold is set to 200rpm and the lowest possible speed is 300 rpm.

 

My MB: supermicro X11SCH-F

 

I'm sorry for my bad english, as this is not my mother language.

 

I hope you can help me.

Thank you!
 

1.JPG

2.JPG

Link to comment
  • 2 weeks later...

Another X570D4U-2L2T user here, trying to get this plugin to play nice. 

@cakes044  Could you please share what you did to ipmi2json to get it working?
Even if we can't get a patch together, I'd rather have a "hack" running on my board than not have anything for the drives. 

Edited by Vendigroth
  • Like 1
Link to comment
3 hours ago, Vendigroth said:

Another X570D4U-2L2T user here, trying to get this plugin to play nice. 

@cakes044  Could you please share what you did to ipmi2json to get it working?
Even if we can't get a patch together, I'd rather have a "hack" running on my board than not have anything for the drives. 

Sorry I never posted my actual solution ages ago, I was actually going to fork the plug-in and submit a PR but I thought the author was going to do it.

 

Ill post my changes that have a working solution for these boards later in the day, essentially it was a change to the ‘ipmi2json’ and the ‘ipmifan’ scripts.

  • Like 3
Link to comment
36 minutes ago, cakes044 said:

Sorry I never posted my actual solution ages ago, I was actually going to fork the plug-in and submit a PR but I thought the author was going to do it.

 

Ill post my changes that have a working solution for these boards later in the day, essentially it was a change to the ‘ipmi2json’ and the ‘ipmifan’ scripts.

 

I'm also an X570D4U-2L2T user and would LOVE to get the fan controller functionality working.  This is a GREAT plugin and I love the unRAID community.  

  • Like 1
Link to comment

Hi All,

 

Here is the forked repo with the details of my "fix", keep in mind this is only for X570D4U and X570D4U-2L2T users, I purposely didn't include this to be backwards compatible because I didn't have those boards to do the proper testing or the time to make this backwards compatible.

 

This is the commit in question that has my "fix", I also fixed some minor bugs with the shell commands:

https://github.com/lucasteligioridis/IPMI-unRAID/commit/49281fd8

 

I can't quite remember if the ipmi2json script works flawlessly or not, I may have hacked in some values on the first run, full disclosure I haven't used that one since I first put it together and used it.

 

But the ipmifan script works wonders with these boards, haven't had a problem with it since implementation.

You essentially want to copy the two files mentioned from here:

https://github.com/lucasteligioridis/IPMI-unRAID/tree/master/source/ipmi/usr/local/emhttp/plugins/ipmi/scripts

 

Into the correct location on the Unraid system, which is here:

- /usr/local/emhttp/plugins/ipmi/scripts

 

You will have to restart the ipmifan service in the Unraid UI and it all should hopefully work, at the very worse the ipmi2json script might not work perfectly as intended, but it will at least generate some of the json correctly. Let me know how it goes, happy to help further. I'm not at home at the moment and don't have SSH access to validate the above just yet.

 

To have this fix "permanent" I just copy the ipmifan script into the above directory whenever the machine boots, because the plugins get reloaded each time Unraid restarts so the old code will be restored. Obviously any updates to the plugin will have the same effect. If I get more time I'd love to fix this into a proper patch and merge it upstream.

Edited by cakes044
Formatting.
  • Thanks 2
Link to comment

Thank you @cakes044! 
I got It working on my X570D4U-2L2T. 
ipmi2json worked when I switched all the fans to manual mode in the BMC. Think $manual_cmd wasn't doing it for me for some reason. Didn't look too hard since there's a workaround. 
I also had to adjust the fan positions in the board.json as they were bumped down by one.

ipmifan  does work great as-is though! Thanks again. 

  • Like 1
Link to comment
On 3/30/2021 at 5:48 PM, DerUnraider said:

The IPMI plugin was working without any issues until some couple of days ago (maybe until updating unraid to latest stable version but I can't confirm this).

The problem is that when the server starts all fans running on full speed and do not change the speed. If I turn off the fan control and turn it on afterwards the fans are running normal for about...~1h and than turning back to full power. I didn't changed anything in bios or any other system related setting. This also happens when the server runs for days.
The threshold is set to 200rpm and the lowest possible speed is 300 rpm.

 

My MB: supermicro X11SCH-F

 

I'm sorry for my bad english, as this is not my mother language.

 

I hope you can help me.

Thank you!

 

The same happened to me this morning. All fans suddenly started to run at full speed although I didn't change anything in the fan control config. There are no fan log entries or higher temperatures occurring at the time it happened.

After turning fan control off and on again all fans are running normal for about 15-35 minutes but then start to rotate with full speed again.

ps -aux | grep ipmifan

shows the php ipmifan script as running during this time.

 

I'm also using a Supermicro X11SCH-LN4F Board which is the same board as yours but with 4 NICs.

Did you find a solution for this or or has it not happened since?

 

 

Edit: I tried to set the fan speed minimum to a higher level to not get a 0 rpm reading from a fan which will result in supermicros fan control take over and speeding up all fans due to a hard-coded failsafe feature in Supermicro boards. Low-rpm fans like Noctua will sometimes report 0 rpm for a short time although they still rotating with very low speed. Restarting the ipmifan script will takeover the fan control from the Supermicros failsafe control. 

Since I haven't re-adjusted the fan speed minimum recently I'm wondering why this happened today?

Nevertheless the fans still speed up and now in an even shorter period. Restarting the server didn't help. Changing the minimum values and clicking on Apply will also reset all fans to normal speed. 

I tried to run the script with --debug but there were not entries during the time the fans speed up.

I suspect the Supermicro failsafe is somehow involved in this but I can't seem to get telegraf to log the rpms in a shorter interval than 1 minute although I set the global metric to 10 second.

 

Edit 2: Like I assumed it was one specific fan that was running too low for a short time. I had to adjust the minimum threshold from approx. 30 to 50 percent until the fan was not reporting 0 rpm after a while.

Since I didn't change any thresholds or minimum I am still wondering why this happens today and there have never been such problems in the last 7 months since the sever has been running.

Perhaps the electronics of the fan have been somewhat affected by the continuous operation? Are there other reasons why something like this could occur?

Edited by kennymc.c
Link to comment
On 4/12/2021 at 5:24 PM, kennymc.c said:

 

The same happened to me this morning. All fans suddenly started to run at full speed although I didn't change anything in the fan control config. There are no fan log entries or higher temperatures occurring at the time it happened.

After turning fan control off and on again all fans are running normal for about 15-35 minutes but then start to rotate with full speed again.


ps -aux | grep ipmifan

shows the php ipmifan script as running during this time.

 

I'm also using a Supermicro X11SCH-LN4F Board which is the same board as yours but with 4 NICs.

Did you find a solution for this or or has it not happened since?

 

 

Edit: I tried to set the fan speed minimum to a higher level to not get a 0 rpm reading from a fan which will result in supermicros fan control take over and speeding up all fans due to a hard-coded failsafe feature in Supermicro boards. Low-rpm fans like Noctua will sometimes report 0 rpm for a short time although they still rotating with very low speed. Restarting the ipmifan script will takeover the fan control from the Supermicros failsafe control. 

Since I haven't re-adjusted the fan speed minimum recently I'm wondering why this happened today?

Nevertheless the fans still speed up and now in an even shorter period. Restarting the server didn't help. Changing the minimum values and clicking on Apply will also reset all fans to normal speed. 

I tried to run the script with --debug but there were not entries during the time the fans speed up.

I suspect the Supermicro failsafe is somehow involved in this but I can't seem to get telegraf to log the rpms in a shorter interval than 1 minute although I set the global metric to 10 second.

 

Edit 2: Like I assumed it was one specific fan that was running too low for a short time. I had to adjust the minimum threshold from approx. 30 to 50 percent until the fan was not reporting 0 rpm after a while.

Since I didn't change any thresholds or minimum I am still wondering why this happens today and there have never been such problems in the last 7 months since the sever has been running.

Perhaps the electronics of the fan have been somewhat affected by the continuous operation? Are there other reasons why something like this could occur?

Problem still exist but I also found that one? fan is reporting 0rpm from time to time and maybe thats the same reason as you reported. How did you change the threshold? By ipmi command or? Actually lowest rpm for the chassis fans are 300 rpm and 1000 for the cpu and yes these are Noctua fans probably the same you have.

Link to comment
3 hours ago, DerUnraider said:

Problem still exist but I also found that one? fan is reporting 0rpm from time to time and maybe thats the same reason as you reported. How did you change the threshold? By ipmi command or? Actually lowest rpm for the chassis fans are 300 rpm and 1000 for the cpu and yes these are Noctua fans probably the same you have.

 

I changed the fan speed minimum on the Fan Control/Fan Settings tab in the IPMI plugin page. When I originally installed my sever I set this to 31.2% since this was the threshold where my fans were not reporting a 0 rpm reading when they are running with their lowest speed.

Noctua fans do not seem to be able to report rpm readings under 300 rpm although they actually might be running slower for a short time. Instead a 0 rpm reading is reported and causing all fans to speed up.

I never saw this issue with one my cpu fans from Scythe which should even be able to be running 50 rpm slower. Maybe this is related to some special voltage tuning from Noctua to reducing noise.

Link to comment

Anyone any ideas on this appearing in syslog .... bear in mind i have only just installed ipmi plugin in anattempt to sort out fan problems on my x10dri-ln4f which seems to have no pwn control that i can latch on to ...

 

'kernel: x86/PAT: ipmi-sensors:62796 map pfn expected mapping type uncached-minus for [mem 0x796aa000-0x796aafff], got write-back'

 

these errors just fill the syslog until the system eventually becomes unresponsive necessitating a reboot. Hve now disable plugin and messages stopped. Any ideas anyone?

Link to comment

Any idea if you can use the configurator on a Dell PowerEdge R730XD?  I have the 10G NIC installed and the fans go absolutely ballistic.  I've ran the manual ipmitools command which drops them down to 9,500 RPM (ish) which is still VERY loud.  I was rather hoping that I could use this tool to set the speeds lower to say like 7-8k RPM.   

I always Get this message when I go to Fan Control Tab.  It tells me to click on configure button but there is no configure button.

Any ideas?

 

Screen Shot 2021-04-15 at 4.06.10 PM.png

Link to comment

So just setting up my new-to-me server with a supermicro X9dax-7f mobo. It has built in power usage tracking via ipmi and I can get the output if I poll it manually:

 

~# ipmitool dcmi power reading

    Instantaneous power reading:                   112 Watts
    Minimum during sampling period:                  9 Watts
    Maximum during sampling period:                406 Watts
    Average power reading over sample period:      126 Watts
    IPMI timestamp:                           04/16/2021 10:16:18 AM CDT    Sampling period:                          00078026 Seconds.
    Power reading state is:                   activated

 

My question is can the Instantaneous power reading (or better yet multiple readings) be output to a sensor reading like the rest? Is there a way to add a manual sensor to poll this information?

 

Ideally I would like to be able to see the power usage along the bottom taskbar but just having a place I could see it would be cool as well.

Link to comment

SM X10DRi-LN4+ board. How do i identify my IPMI address? Been into bios, but seems it needs a mac address to go alongside my allocating a static IP, does that sound about right? Cant see a mac address anywhere though ...  whats the procedure for setting things up?

Edited by superloopy1
Link to comment

there is a dos supermicro ipmi tool you can use that can change the ipmi ip and a bunch of other settings.

 

The ip address is generally set in the bios. On my X9 board it has a setting to change the IP and it will use DHCP to get a ip and then set that as static.

 

If you want a custom static address, my guess is that ipmi tool from supermicro.

Link to comment

Anyone able to help me set up my ipmi? I can't identify the ipmi Mac address so don't see a way if assigning a local IP without this. Board is an X10DRI-LN4+ with ipmi. Is it wise to reset the BMC itself which appears to be the only other option in the bios? Will this affect the running of my server in any way or is the BMC independent? Still learning about these SM boards and, like others, need to get control of the fan's back 'in house' ... thx!

Link to comment
2 hours ago, superloopy1 said:

Anyone able to help me set up my ipmi? I can't identify the ipmi Mac address so don't see a way if assigning a local IP without this. Board is an X10DRI-LN4+ with ipmi. Is it wise to reset the BMC itself which appears to be the only other option in the bios? Will this affect the running of my server in any way or is the BMC independent? Still learning about these SM boards and, like others, need to get control of the fan's back 'in house' ... thx!

Mine is printed on a sticker on the motherboard. IPMI xxxxxxxxxxxx. Right next to the sticker with serial number and the sticker with the Mac for the other 2 nics.

Link to comment
25 minutes ago, wgstarks said:

Mine is printed on a sticker on the motherboard. IPMI xxxxxxxxxxxx. Right next to the sticker with serial number and the sticker with the Mac for the other 2 nics.

Thankyou!

 

I'll open things up and take a look. If i remember there was a manufacturer sticker on there. Any idea of whther i'm safe to reset BMC without it affecting anything else, i'm becoming paranoid. I cant get the parity to stick, disks to spin down and now the fans!!

Link to comment
3 hours ago, superloopy1 said:

Anyone able to help me set up my ipmi? I can't identify the ipmi Mac address so don't see a way if assigning a local IP without this. Board is an X10DRI-LN4+ with ipmi. Is it wise to reset the BMC itself which appears to be the only other option in the bios? Will this affect the running of my server in any way or is the BMC independent? Still learning about these SM boards and, like others, need to get control of the fan's back 'in house' ... thx!

 

I have an ASRock Rack board and mine displays the IPMI/BMC IP address as soon as I power up and before the system completes the POST to access BIOS.  I can then look for the MAC in my DHCP.  If you are not able to do that and you have a managed switch, you should be able to SSH into your switch and use commands similar to "show arp" and/or "show mac-address" (google documentation specific to your switch) to identify the MAC addresses by port.  If that is not an option for you then you could also try a network port scanner to find all active IP addresses on your network and their associated MAC addresses (https://www.softperfect.com/products/networkscanner/).  I've used this one many times over the years with great success.  

 

Resetting your BMC should have no affect on your server's functionality.  If your BMC was set with a static IP that is not part of your existing subnet then doing the reset will be the only way to get it back to defaults which should set the network interface to use DHCP.   Any of the methods I outlined above will not work if the IP was set to static outside of your existing network scope.  The only thing that might work is if your system displays the IPMI/BMC IP address pre-POST and you added an IP address to your client computer that was in that subnet scope.  Example: Your home network is 192.168.0.x 255.255.255.0 and the BMC was set to 10.0.0.86 255.255.255.0.  You could add a second IP address to your computer's NIC by accessing the adapter's properties and setting your IP manually to what your current address is so you maintain connectivity to your existing network, and add a second IP address that would be 10.0.0.87 255.255.255.0 (not the same as the IP of the BMC, but on the same subnet) and then you should be able to access the web interface of the BMC by going to the BMC's IP address in your web browser, log in and reset the settings.  But as a side note, if it has a static IP then it probably has a non-default password and your only recourse will be to reset the BMC in BIOS.  

Link to comment
9 minutes ago, superloopy1 said:

Thankyou!

 

I'll open things up and take a look. If i remember there was a manufacturer sticker on there. Any idea of whther i'm safe to reset BMC without it affecting anything else, i'm becoming paranoid. I cant get the parity to stick, disks to spin down and now the fans!!

 

BMC reset should have little or nothing to do with your unRAID server performance.  It designed to access and manage your hardware associated with your motherboard, monitor onboard sensors, set manual fan curves, update BIOS, and remote console level access.  Don't fret over a reset to defaults.  

 

Spin down of drives has several factors that are going to play into it.  If you are using any SAS drives you absolutely must install the SAS drive plugin (from the apps tab).  This is a known issue with SAS drives and things are getting better, but you should still use the plugin until there is better native support of SAS drive management.  If you are not using SAS, then you should take a look at your Drive Settings to validate/update your spin down wait time.  Also, are you using the Turbo Write plugin?  This will affect drive spin up.  What is your mover schedule?  Are you using the Mover Tuning plugin?  If you have a smaller cache, and have mover scheduled to run frequently to keep the cache from filling up, and you are doing lots of writes to your cache, then I would have at a minimum of 1 hour wait time on my spin down, with mover running every hour and turbo mode enabled for mover.  Disks spinning at low speed with minimal disk activity is better for the life of the disk then lots of spin ups and spin downs.  If disk life isn't a concern and you just want minimum power usage, then shorter wait times are better for your situation.  Also, you may consider not using cache for those shares and have them write direct to disk.  Then you can set your mover to run less frequently for those shares you need fast write speeds going to long term disk storage.  These are all things to consider in building out your storage and performance plan. 

 

"I cant get the parity to stick" is a bit more concerning and very unclear as to what you mean.  If you haven't yet, start a separate thread for this issue in the General support forum as this would be completely unrelated to the IPMI plugin and absolutely post with diagnostics so people don't have to ask for them to assist you.  If your system is locking up or crashing before a parity check can complete then you need to take a step back, do hardware testing to find and isolate the issue, then slowly build upon a stable and working platform.  

Link to comment

So I'm not gonna lie, I have not read all 56 pages of this thread. :)
I have a Dell Poweredge T420 (12th gen).

Is there any chance of this plugin controlling the fans, or is it still only for those 2 boards mentioned in the OP?

EDIT: Also, I'm doing something wrong, since when I run any "ipmitool" command from the terminal, I get the output "bash: ipmitool: command not found". 

What key step have I missed?

Edited by Elmojo
Link to comment

Many thanks to @wgstarks @TexasDaddy for their assistance. Embarassingl for me, on opening my case to look for BMC mac address 'sticker' i found that IPMI jumper setting was set to 'disabled' (why oh why!) so when i moved the shunt everything was revealed, voila!! An easy mistake to make, oops!

 

So ... with ipmi set up correctly ive moved on to fan control via this plugin and i havent a clue. Anyone care to post up a couple of screenshots of their settings on a SM x10 board preferably? I think i've managed to get through the screens but i fall down when it comes to the fan control config editor which when i pick 'sensors' from the dropdown list shows absolutely nothing to edit, should it? Or do i have to populate an empty base file. If so, how?

 

As my board has dual cpus how do these work alongside this plugin? There are nine fan inputs (1-6, A-C) with two currently unused but to be allocated to onboard coolers as a second fan on each.

 

Assistance so far has been invaluable, thanks.

Edited by superloopy1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.