UPS Failing? - Unclean shutdown after power outage

[JP]

I've had two recent power outages and both have seemingly resulted in unclean shutdowns from unraid.  I know there are a few things I could test like:  1)  Will it shutdown properly at all?  I'm going to test this once my parity check is complete.  2)  Connect something like only a light to one of the protected outlets on the UPS and trip the fuse.  This is only a guess right now, but I'm wondering if the UPS ability to catch the power outage in time is failing and this might help me answer that.  If the light shuts off with a power outage then it seems like that would be the culprit.

 

However, in the meantime and while I'm waiting for the parity check, I thought I would ask whether there would be anything in the diagnostics that might tell me what the root cause is here?  I'm not great looking at this information, but I tried myself and it seems like the diagnostics only pick up at reboot, everything before that is lost.  Is that accurate?  

 

Thanks for any help.

tower-diagnostics-20240802-1207.zip

[JonathanM]

50 minutes ago, JP said:

2)  Connect something like only a light to one of the protected outlets on the UPS and trip the fuse.

If you can manage to simultaneously feed your Unraid uninterrupted power, then when the UPS goes into backup mode Unraid should start the shutdown process.

 

If you connect something with a similar electrical load to the UPS, maybe a small space heater, then you can see if the Unraid shutdown beats the UPS running out of power.

 

It's entirely possible Unraid's shutdown process is timing out before something can get stopped, forcing Unraid to shut down uncleanly. If you are running VM's, they may not be stopping properly, or take too much time. Personally I run apcupsd software inside my guest VM's in slave mode, so when the host detects a power outage the VM's immediately start the shutdown process.

[JP]

4 minutes ago, JonathanM said:

If you can manage to simultaneously feed your Unraid uninterrupted power, then when the UPS goes into backup mode Unraid should start the shutdown process.

 

If you connect something with a similar electrical load to the UPS, maybe a small space heater, then you can see if the Unraid shutdown beats the UPS running out of power.

 

It's entirely possible Unraid's shutdown process is timing out before something can get stopped, forcing Unraid to shut down uncleanly. If you are running VM's, they may not be stopping properly, or take too much time. Personally I run apcupsd software inside my guest VM's in slave mode, so when the host detects a power outage the VM's immediately start the shutdown process.

 

Yea, I've got some testing to do.  Another 8 hours until the parity check is done.  So, there is nothing in the diagnostics that can help tell us what might have happened here?

 

Just a guess right now, but I don't think it is the battery.  This server doesn't pull a big load at all.  It is doing a parity check with all drives spinning right now and the load isn't even breaking 100 W.  The runtime left on the batter is 43 minutes at this load, which seems like plenty.  Here is how I have my settings configured:

 

 

I'm assuming the first two line items are leveraged when whichever one comes first.  Also, another reason I don't think it is the battery, is the UPS still had 50% battery left when I checked it during the power outage.  BUT it was not ON.  I had to power cycle it to see it was at 50% and as you can see from my settings, it is not supposed to shutdown the UPS after SHUTDOWN.  Sort of further leading me to believe that when the power outage hit, it shut down everything, the UPS and the UNRAID server all at once, not what the UPS should be doing.  Again, just a guess for now, but I'll provide more information later.   

[itimpi]

3 hours ago, JP said:

seems like the diagnostics only pick up at reboot, everything before that is lost.  Is that accurate

As you thought, the syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

[Vr2Io]

It still look like battery problem, mainly because all have shutdown suddenly.

 

Your setting ( 3rd line ) were shutdown immediately when on battery, that means UPS haven't provided or just very short time protection.

On 8/3/2024 at 2:23 AM, JP said:

The runtime left on the batter is 43 minutes at this load

Those are base on the pre defined "normal" battery case, UPS really won't know battery healh or not, it just base on battery voltage and estimate the runtime for you.

[JonathanM]

On 8/2/2024 at 2:23 PM, JP said:

Just a guess right now, but I don't think it is the battery. 

You MUST load test to find that out. Batteries can (and often do) fail in a state where voltage is normal during charging, but as soon as a load is applied, the battery falls over almost immediately.

[JP]

23 hours ago, itimpi said:

As you thought, the syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

Thanks for this.  It would really be helpful.  I'll look into it more once I get the UPS figured out and have the server up and running again.

[JP]

1 hour ago, Vr2Io said:

It still look like battery problem, mainly because all have shutdown suddenly.

 

1 hour ago, JonathanM said:

You MUST load test to find that out. Batteries can (and often do) fail in a state where voltage is normal during charging, but as soon as a load is applied, the battery falls over almost immediately.

 

Yea, spot on from both of you.  Thank you.  I took the UPS out and started to test with a lamp and nothing seemed wrong at first.  I thought the load might not be enough so I tried to think of what energy consuming devices I might have around.  Then I thought of this candle warmer my wife has always had that I hate.  It puts out a ridiculous amount of heat (usually when the A/C is on 😡) so I thought I might add that.  Sure enough the battery went from full, down to about 10% in one minute.  Definitely seems to be the battery.  I've purchased a replacement battery and it will be here tomorrow.  I'm optimistic this will take care of it, but I'll report back once I know for sure.  

[JP]

 

On 8/3/2024 at 3:56 PM, JP said:

I've purchased a replacement battery and it will be here tomorrow.  I'm optimistic this will take care of it, but I'll report back once I know for sure.  

I really can't say enough good things about Cyberpower.  I bought this UPS literally a decade ago.  It has worked flawlessly until now and the culprit was simply the battery, a $45 replacement no less.  I'm sort of in disbelief I got 10 years out of the original battery. 

 

Anyway, replaced the battery and charged it for 8 hours as the directions said.  I then spun up all the drives and pulled the power cable from the UPS.  The battery and UPS held and ramped down power slowly as it should.  It got to 75% left on the battery and Unraid cleanly shutdown as it should.  It is so nice when things just work.

 

Historically, I've preferred having the Unraid server just go completely shutdown and stay there after a power failure because I like to know it happened, but I'm starting to feel a little more comfortable automating it and might set it up so that Unraid boots back up and starts the array on its own once power is restored.  I think I have to do something in the BIOS and in the UNRAID settings to accomplish this, but not sure.  I'll be looking into it more.  Thanks again for the help here. 

[Vr2Io]

49 minutes ago, JP said:

$45 replacement no less.

Two or one 12v battery ? If two, quite reasonable.

 

49 minutes ago, JP said:

got 10 years out of the original battery. 

If life reach 5yrs it almost excellent, my previous bad experience even less then 3 months. So I have a big project which change to use LFP battery. Of course, I greatly modifiy the UPS, make it green and reliable.

 

Below are the battery pack which use and  already cut the UPS in half to remove battery area.

 

 

 

Edited August 6 by Vr2Io

[JP]

46 minutes ago, Vr2Io said:

Two or one 12v battery ? If two, quite reasonable.

It was one battery.  I could have gotten a cheaper battery than Cyberpower, but at $45, I thought that was still cheap and I felt better staying with Cyberpower.  

 

Your battery looks cooler than my car.   Those terminals look beastly.  Nice work!

[Vr2Io]

I consider change LFP or not for longtime, the cost about $190 (24v 50ah), if continue use PB battery it cost $154 (12v 17ah x2) and PB can't guarantee life time.

 

Above is the main reason for me to try LFP battery, but LFP have higher firing risk this is the only concern factor.

Edited August 6 by Vr2Io