Unclean shutdown with Cyberpower UPS AVR

[dtbingle]

My unRAID box is hooked up to a CyberPower 1500avr UPS.  Within unRaid settings, it's set to shutdown at 25% remaining or 15 minutes.  However, when power to the UPS is lost, it seems unRAID performs actions immediately - even though the UPS power is at 100% with 150 min+ of estimated time left.

 

The strange thing is, when the power comes back and I go to the main unRAID status page on my main PC, it states there was an unclean shutdown and the disks are no longer mounted (and initiates a parity check).  The other weird part is that I never had to press the power button on the unRAID PC case to turn the unit back on.  It's almost as if the box stays powered up, but uncleanly unmounts the disks as soon as a loss of power is detected at the UPS.

 

Any ideas to resolve this?

[Frank1940]

First thing, what is your setting for "Time on battery before shutdown (seconds):"?

 

Second thing, how old is your battery?  Those things have a very definite life!  And you really can't depend on that "Runtime Left" number. 

 

I would suggest that you run some tests on the UPS.  You can do this with a dummy load (PREFERRED- say four hundred watts of light bulbs) or your server with the array stopped.  Pull the plug on the UPS and see how long the UPS runs.  You might be surprised to find out that it may not last the fifteen minutes  The same batteries are often used in UPS of different AV ratings, The runtime at full load is simply significantly shorter for the high rated units! If it does not run at least two minutes, you need to replace the batteries. 

 

The battery has (1) be able to supply the peak load when all of the HD's are spun up for the array shutdown and (2) to last at least long enough to complete the array shutdown, write out the super.dat file (which contains a flag to indicate a successful shutdown), and signal the server's PS to shutdown.  On many servers, Dockers and VM's will also have to be stopped and this has caused problems in the past if they 'hang'. 

 

 

[dtbingle]

First thing, what is your setting for "Time on battery before shutdown (seconds):"?

 

Second thing, how old is your battery?  Those things have a very definite life!  And you really can't depend on that "Runtime Left" number. 

 

I would suggest that you run some tests on the UPS.  You can do this with a dummy load (PREFERRED- say four hundred watts of light bulbs) or your server with the array stopped.  Pull the plug on the UPS and see how long the UPS runs.  You might be surprised to find out that it may not last the fifteen minutes  The same batteries are often used in UPS of different AV ratings, The runtime at full load is simply significantly shorter for the high rated units! If it does not run at least two minutes, you need to replace the batteries. 

 

The battery has (1) be able to supply the peak load when all of the HD's are spun up for the array shutdown and (2) to last at least long enough to complete the array shutdown, write out the super.dat file (which contains a flag to indicate a successful shutdown), and signal the server's PS to shutdown.  On many servers, Dockers and VM's will also have to be stopped and this has caused problems in the past if they 'hang'.

 

I'll have to check on the 'seconds' setting when I get home.  If it's set to something super low where it basically shuts down immediately on loss of power, it should still perform a clean shutdown, right?

 

As for the UPS, it's only about 9 months old.  If I had to place a bet on it, I'm pretty sure the UPS battery condition is not the problem.  The most recent time this happened I was doing electrical work and shut off the main power.  The power was off for about an hour and a half with only the unRAID box on (45 watt draw while idle).  After the ~90 minutes, I booted my main PC which is also on the UPS and ran for 10-15 minutes while checking the unRAID status.  Over this time, the UPS never ran out of juice and the unRAID box never 'powered off', but the array was unmounted.

 

Since you mentioned VMs and Dockers......the only docker I have is Plex Media Server.  How would I be able to tell if this was causing the hangup?

[Frank1940]

 

Since you mentioned VMs and Dockers......the only docker I have is Plex Media Server.  How would I be able to tell if this was causing the hangup?

 

Stop the array and power down the system from the GUI.  If your Docker isn't causing an issue, the server will complete the shutdown quickly.  If it is 'hanging', it may never shutdown.  The time required for the entire process (stopping the array and powering down the system) will give you a good feel on how long the UPS has to be able to last once the apcupsd starts the actual shutdown after any delays you have set up.

[Frank1940]

 

As for the UPS, it's only about 9 months old.  If I had to place a bet on it, I'm pretty sure the UPS battery condition is not the problem.  The most recent time this happened I was doing electrical work and shut off the main power.  The power was off for about an hour and a half with only the unRAID box on (45 watt draw while idle).  After the ~90 minutes, I booted my main PC which is also on the UPS and ran for 10-15 minutes while checking the unRAID status.  Over this time, the UPS never ran out of juice and the unRAID box never 'powered off', but the array was unmounted.

 

 

So everything worked correctly while you had the power service turned off for ninety minutes!  From your description, I assume that a shutdown was never called for and the server did not actually shutdown. 

 

You say that you also have a PC on this UPS.  How are you triggering it to shutdown? 

 

I am also assuming that the UPS plugin on your unRAID server is actually controlling entire process.  I would suggest setting the 'Time on battery before shutdown (seconds):' to about 30 seconds and test the system to see if your unRAID server actually is going through the proper shutdown sequence when the UPS power plug is pulled.  I would further suggest you watch on a monitor attached to the server to see that things are happening properly.

 

BTW, I have my time on battery set to 30 seconds as I have found that if my power is out that long, it will be out for a minimum of four hours.  (some drunk has probably snapped a power pole in two as the best case scenario.)  You have remember that discharging your UPS battery is actually very hard on its life and if you do discharge it, then you should wait for a minimum of four to eight hours after the power comes back on to allow time for the battery to recharge in case you have a second outage.  Setting the time on battery to a fairly short period addresses both of these issues.

 

[dtbingle]

Stop the array and power down the system from the GUI.  If your Docker isn't causing an issue, the server will complete the shutdown quickly.  If it is 'hanging', it may never shutdown.  The time required for the entire process (stopping the array and powering down the system) will give you a good feel on how long the UPS has to be able to last once the apcupsd starts the actual shutdown after any delays you have set up.

 

Ahhh okay.  So from this statement, I take it that dockers/VMs continue running after the array has been unmounted.  My assumption is that the UPS sends the shutdown signal, arrays unmount, but the docker hangs and the flag for a clean shutdown is never set and the unit never actually powers off.

 

 

 

So everything worked correctly while you had the power service turned off for ninety minutes!  From your description, I assume that a shutdown was never called for and the server did not actually shutdown. 

 

You say that you also have a PC on this UPS.  How are you triggering it to shutdown? 

 

I am also assuming that the UPS plugin on your unRAID server is actually controlling entire process.  I would suggest setting the 'Time on battery before shutdown (seconds):' to about 30 seconds and test the system to see if your unRAID server actually is going through the proper shutdown sequence when the UPS power plug is pulled.  I would further suggest you watch on a monitor attached to the server to see that things are happening properly.

 

 

Sorry, I was a bit unclear.  The unRAID box was running for the 90 minutes or so and never fully shutdown/turned the power off.  However, when supply source for the UPS was turned back on (with the unRAID box still running and never shut off), I checked the unRAID status via GUI from another PC.  At this time, I could see that the array was unmounted, with the important detail that I never had to press the physical power button to start it back up.

 

 

BTW, I have my time on battery set to 30 seconds as I have found that if my power is out that long, it will be out for a minimum of four hours.  (some drunk has probably snapped a power pole in two as the best case scenario.)  You have remember that discharging your UPS battery is actually very hard on its life and if you do discharge it, then you should wait for a minimum of four to eight hours after the power comes back on to allow time for the battery to recharge in case you have a second outage.  Setting the time on battery to a fairly short period addresses both of these issues.

 

 

Good to know!  Never really considered the discharge cycles on the UPS.  I might change it to a 5 or 10 minute shutdown.  Typical outages for me are split 50/50 with flickering power with on/off every few minutes or the extended 4 hour+ session

[Frank1940]

Understand that the UPS shutdown does the same things that you do when you shut the server using the GUI.  In fact, apcupsd calls the same software packages as you are  using.  They only real difference is that it is a script initiating the process rather than you (one of the true advantages of Linux/UNIX!). 

 

Everyone's situation is a bit different as to type of outage that they see on their area.  Five or ten minutes is as reasonable as I find 30 seconds to be  in my situation.  The quick 1-5 seconds outages that reset every digital clock in the house are handle gracefully...

[aptalca]

Troubleshooting UPS issues are really annoying and involve having to sit in front of the server, watching it like a hawk for hours.

 

I have the Cyberpower 1000 AVR LCD and although it works with the apcupsd that is built into unraid, the working functions are rather basic. And I'm having a huge issue.

 

Before I get into that, I suggest you look into various settings before you try and troubleshoot your issue.

 

1) Check the bios for settings related to power loss. Some bios will let the computer automatically boot up after power restore (or go to previous state, if off prior to power loss, it will remain off after restore, but if on, it will turn on, depending on the bios). Make sure you turn it off otherwise you can't tell whether the computer was previously shut down or not unless you physically watch it non-stop.

2) In unraid settings, set the "Time on battery before shutdown (seconds)" to something rather short like 60 seconds. That is the most reliable setting as it does not vary. "Runtime left to initiate shutdown (minutes)" and "Battery level to initiate shutdown (%)" settings are not accurate and may vary with battery age (they are both estimates based on current voltage readings).

3) Install the powerdown plugin. It does a better job of making sure no process get stuck and prevents shutdown

 

Now my issues were hard to figure out but boiled down to two things:

1) The apcupsd plugin in unraid is unable to shut down cyberpower UPSes (This is the feature that makes sure the UPS batteries aren't depleted). Unraid sends the kill signal, but Cyberpower remains on. In that case, there is no point in having the server start automatically on power restore. Because the UPS power will only cut off when it runs out of battery, you do not want your server to start up when power comes back and your UPS is trying to charge its batteries from empty.

2) (Major issue) When my cyberpower is charging its batteries, the moment they get full and the UPS switches from charging to not charging, it cuts the power to the devices for a split second. It causes an unclean shutdown. No clue why. But there were other reports with Cyberpower UPSes here, so it may be an issue with a product line, or may be due to some incompatibility with the apcupsd plugin, not sure.

 

The second issue was a real pain to figure out. What I witnessed was the following:

1) Power outage happens, lasts for 30 min

2) Server shuts down after 5 min

3) UPS remains on, depleting batteries at a slow rate.

4) When power is restored, I turn the server on, UPS battery is at something like 60%, all looking good.

5) I check after a few hours and my server is off. Upon start, it initiates a parity check

 

What really was happening was that between 4 and 5, once the batteries were fully charged, it would cut the power to the server for a split second. Literally the last thing you would expect from a UPS. I only figured it out with the lamp test. I drained the batteries to 60% with a floor lamp. Turned the server on, but stopped the array and waited. Sure enough, once the battery level reached 100%, server shut off.

 

So what I do now is that if there is a power outage, I make sure that the UPS is fully charged before I turn the server back on.

 

I don't really recommend a cyberpower for use with unraid. I would get an APC myself if I was having more frequent power outages (I have maybe 1 per year and it only lasts for a few minutes)

[Frank1940]

I think that are problems with a lot of individual model/series of UPS's.  For example, I have a note on the side of my APC unit that I use on my main Media server that say "Be sure that the UPS is running on Line before starting server".  (I have the apcupsd set up to shut the UPS down as the last thing it does after a power outage.)  The reason for this note is that model line for some reason does a testing sequence as it starts up.  First, it run on battery to check the inverter circuit.  That is does some sort of function checking of its various features.  Then it shuts down the inverter circuit for about two seconds.  (If I have started the server, this is more than enough time to crash it!!!)  The testing now completed, it shifts to 'On Line' operation and the @#&% thing works the way any good UPS should. 

 

So CyberPower is not the only manufacturer to have units that can drive one to drink... 

[garycase]

Agree -- every unit seems to have its eccentricities.  I have both APC and CyberPower units on my various systems (along with an old Belkin that's still working just fine).    My CyberPower unit DOES turn itself off when instructed to ... but this is on a PC desktop using the PowerPanel software.  My two main UnRAID servers are both on APC units, but I do have a test server with a CyberPower => I'll have to test it and see if it has the same issue you've outlined (i.e. not turning off the UPS).

 

I definitely agree that you do NOT want to leave your server running for any significant amount of time on battery, regardless of what the reported runtime is.    I have all of my systems set to shut down after 5 minutes.

 

 

[dtbingle]

Troubleshooting UPS issues are really annoying and involve having to sit in front of the server, watching it like a hawk for hours.

 

I have the Cyberpower 1000 AVR LCD and although it works with the apcupsd that is built into unraid, the working functions are rather basic. And I'm having a huge issue.

 

Before I get into that, I suggest you look into various settings before you try and troubleshoot your issue.

 

1) Check the bios for settings related to power loss. Some bios will let the computer automatically boot up after power restore (or go to previous state, if off prior to power loss, it will remain off after restore, but if on, it will turn on, depending on the bios). Make sure you turn it off otherwise you can't tell whether the computer was previously shut down or not unless you physically watch it non-stop.

2) In unraid settings, set the "Time on battery before shutdown (seconds)" to something rather short like 60 seconds. That is the most reliable setting as it does not vary. "Runtime left to initiate shutdown (minutes)" and "Battery level to initiate shutdown (%)" settings are not accurate and may vary with battery age (they are both estimates based on current voltage readings).

3) Install the powerdown plugin. It does a better job of making sure no process get stuck and prevents shutdown

 

Now my issues were hard to figure out but boiled down to two things:

1) The apcupsd plugin in unraid is unable to shut down cyberpower UPSes (This is the feature that makes sure the UPS batteries aren't depleted). Unraid sends the kill signal, but Cyberpower remains on. In that case, there is no point in having the server start automatically on power restore. Because the UPS power will only cut off when it runs out of battery, you do not want your server to start up when power comes back and your UPS is trying to charge its batteries from empty.

2) (Major issue) When my cyberpower is charging its batteries, the moment they get full and the UPS switches from charging to not charging, it cuts the power to the devices for a split second. It causes an unclean shutdown. No clue why. But there were other reports with Cyberpower UPSes here, so it may be an issue with a product line, or may be due to some incompatibility with the apcupsd plugin, not sure.

 

Wow, way more to take a look at than expected haha.  I messed around with it some more yesterday night.

 

1)  Stopping the array and executing 'powerdown' at the command line worked fine - no parity check

2)  When using the "Time on battery before shutdown (seconds)" at 30 seconds, the box shut down (killed powered and all), worked fine - no parity check

3)  I had the UPS shutdown feature enabled before, but what happened in my case is that is seemed like the UPS just turned off before unRAID was fully shut down/powered off.  Maybe there's a weird issue with using the "minutes remaining" and "% left" with certain hardware/UPS combos?

 

I'll check the things you mentioned and try to test your #2 problem as well. 

[garycase]

FWIW I've never seen the "#2 problem"  (an outage caused by reaching full charge) on ANY of the various UPS units I've used -- several APC units, a couple of Belkins, and several CyberPower units, which includes both pure sine-wave models and a couple that only have AVR and stepped waveforms.

 

I wonder if that issue isn't actually with the UPS unit, but with some interaction between APCUPSD and the UPS when the full charge state is recognized.  That would be easy to test -- ASSUMING the problem is consistent on the system that has it => just discharge the UPS a bit; then, while it's charging, unplug the USB communications cable.  Then there won't be any communications with the UPS ... so see if the problem happens when the UPS is fully charged.

 

[aptalca]

FWIW I've never seen the "#2 problem"  (an outage caused by reaching full charge) on ANY of the various UPS units I've used -- several APC units, a couple of Belkins, and several CyberPower units, which includes both pure sine-wave models and a couple that only have AVR and stepped waveforms.

 

I wonder if that issue isn't actually with the UPS unit, but with some interaction between APCUPSD and the UPS when the full charge state is recognized.  That would be easy to test -- ASSUMING the problem is consistent on the system that has it => just discharge the UPS a bit; then, while it's charging, unplug the USB communications cable.  Then there won't be any communications with the UPS ... so see if the problem happens when the UPS is fully charged.

Ugh, you're asking me to do another test? the issue is that my ups is now over 4 years old and the battery indicator is no longer very accurate. While discharging, it stays at 100% for a while and then drops to 65%. And then recharging to 100% takes a long time. So each new test takes several hours.

 

I'll see if I can build enough patience lol

[MSattler]

So what are you guys using for your config file?

 

I am now running 2 CyberPower 1500's.  One for my Emby Server, ESXi Host, and router, another for 2 unRaid servers and a Cisco switch.

 

I'd like the one 1500 to shut down both unraid servers.  Since there is only one USB cable I figured the unraid server I connect the cable to will have a script in it's shutdown script to shut the other server down as well?

 

thanks!

[Frank1940]

So what are you guys using for your config file?

 

I am now running 2 CyberPower 1500's.  One for my Emby Server, ESXi Host, and router, another for 2 unRaid servers and a Cisco switch.

 

I'd like the one 1500 to shut down both unraid servers.  Since there is only one USB cable I figured the unraid server I connect the cable to will have a script in it's shutdown script to shut the other server down as well?

 

thanks!

 

I am going to give a quick (incomplete) answer.  There is a way to tie two unRAID server together so that one UPS can shut both of them down properly.  I seem to recall that it uses the Network connection to handle the signaling between the server.  (This means you have to keep the network up also!)

 

I would suggest that you begin searching in this section of the forum for the details: 

 

          http://lime-technology.com/forum/index.php?board=3.0

 

If you don't find the answer there, I would also suggest that you start your own thread as hijacking this one will not bring your question to very many folks.

[aptalca]

So what are you guys using for your config file?

 

I am now running 2 CyberPower 1500's.  One for my Emby Server, ESXi Host, and router, another for 2 unRaid servers and a Cisco switch.

 

I'd like the one 1500 to shut down both unraid servers.  Since there is only one USB cable I figured the unraid server I connect the cable to will have a script in it's shutdown script to shut the other server down as well?

 

thanks!

 

You need a master/slave setup.

 

Your primary unraid box will be connected to the ups via usb, be set up the regular way and will be the master

 

On the secondary unraid server (slave), in the apcupsd settings, select "ether" for ups cable, "net" for ups type and enter the primary unraid server's ip with the port 3551 (192.168.X.X:3551) into the device field.

 

That way your secondary unraid server will query your primary server for the ups status. Make sure that the router/switch is also on a ups so that when the power goes out, the secondary server can still connect to the primary over lan

[ezhik]

Yep.

 

I have mine setup that way, but you want to make sure your switch is connected to the battery as well so that the server can receive the shutdown request.

 

 

 

-- MASTER --

 

 

 

-- SLAVE --

 

 

 

[JonathanM]

Yep.

 

I have mine setup that way, but you want to make sure your switch is connected to the battery as well so that the server can receive the shutdown request.

Also make sure that the slave shuts down BEFORE the master is scheduled to shut down. I'd leave a pretty good margin so the slave can be fully shut down before the master begins shutdown. If both systems are fully spun down when you have an outage, you could possibly overload the UPS with the momentary surge of both systems spinning up all their drives at once to start the shutdown process.

[ezhik]

Agreed.