[SOLVED] Power Failure

[a12vman]

I had a power failure today but my syslogs show a battery minimum was hit almost immediately. Then it says the power came back on. Here is an excerpt from today(june 16). The only parameter I have in UPS Settings is Battery % to Initiate Shutdown = 60%.  Why is this happening?  I am not convinced there was a power outage today my stove and microwave clocks still show the time.  UPS reports 100% charge wih ~ 43 Minutes of run time.  I manually powered the server back up ~ 6:30pm and everything seems fine so far.

 

Jun 16 14:18:32 MediaTower apcupsd[15730]: Power failure.
Jun 16 14:18:37 MediaTower apcupsd[15730]: Reached remaining time percentage limit on batteries.
Jun 16 14:18:37 MediaTower apcupsd[15730]: Initiating system shutdown!
Jun 16 14:18:37 MediaTower apcupsd[15730]: User logins prohibited
Jun 16 14:18:37 MediaTower root: /usr/local/sbin/powerdown has been deprecated
Jun 16 14:18:37 MediaTower init: Switching to runlevel: 0
Jun 16 14:18:37 MediaTower init: Trying to re-exec init
Jun 16 14:18:37 MediaTower apcupsd[15730]: Power is back. UPS running on mains.
Jun 16 14:18:37 MediaTower apcupsd[15730]: Allowing logins
Jun 16 14:18:49 MediaTower kernel: mdcmd (51): nocheck cancel
Jun 16 14:18:50 MediaTower emhttpd: Spinning up all drives...
Jun 16 14:18:50 MediaTower emhttpd: shcmd (172794): /usr/sbin/hdparm -S0 /dev/sdi
Jun 16 14:18:50 MediaTower kernel: mdcmd (52): spinup 0
Jun 16 14:18:50 MediaTower kernel: mdcmd (53): spinup 1
Jun 16 14:18:50 MediaTower kernel: mdcmd (54): spinup 2
Jun 16 14:18:50 MediaTower kernel: mdcmd (55): spinup 3
Jun 16 14:18:50 MediaTower kernel: mdcmd (56): spinup 4
Jun 16 14:18:50 MediaTower root: 
Jun 16 14:18:50 MediaTower root: /dev/sdi:
Jun 16 14:18:50 MediaTower root:  setting standby to 0 (off)
Jun 16 14:18:53 MediaTower emhttpd: Stopping services...
Jun 16 14:18:53 MediaTower emhttpd: shcmd (172796): /etc/rc.d/rc.libvirt stop
Jun 16 14:18:55 MediaTower root: Domain 992b2eed-7034-08a5-5624-8a8ad130efd9 is being shutdown

 

 

SysLog-20200616.txt

Edited June 17, 2020 by a12vman

[a12vman]

I could really use some help here.  Same issue again this afternoon, here are the bits from SysLog.txt:

 

Jul  7 15:59:28 MediaTower apcupsd[14039]: Power failure.
Jul  7 15:59:33 MediaTower apcupsd[14039]: Reached remaining time percentage limit on batteries.
Jul  7 15:59:33 MediaTower apcupsd[14039]: Initiating system shutdown!
Jul  7 15:59:33 MediaTower apcupsd[14039]: User logins prohibited
Jul  7 15:59:33 MediaTower root: /usr/local/sbin/powerdown has been deprecated
Jul  7 15:59:33 MediaTower init: Switching to runlevel: 0
Jul  7 15:59:33 MediaTower apcupsd[14039]: Power is back. UPS running on mains.
Jul  7 15:59:33 MediaTower apcupsd[14039]: Allowing logins
Jul  7 15:59:33 MediaTower init: Trying to re-exec init
Jul  7 15:59:34 MediaTower nginx: 2020/07/07 15:59:34 [alert] 9175#9175: *3908688 open socket #26 left in connection 9
Jul  7 15:59:34 MediaTower nginx: 2020/07/07 15:59:34 [alert] 9175#9175: *3908686 open socket #19 left in connection 15
Jul  7 15:59:34 MediaTower nginx: 2020/07/07 15:59:34 [alert] 9175#9175: aborting
Jul  7 15:59:37 MediaTower kernel: mdcmd (46): nocheck cancel
Jul  7 15:59:38 MediaTower emhttpd: Spinning up all drives...
Jul  7 15:59:38 MediaTower emhttpd: shcmd (60463): /usr/sbin/hdparm -S0 /dev/sdi
Jul  7 15:59:38 MediaTower kernel: mdcmd (47): spinup 0
Jul  7 15:59:38 MediaTower kernel: mdcmd (48): spinup 1
Jul  7 15:59:38 MediaTower kernel: mdcmd (49): spinup 2
Jul  7 15:59:38 MediaTower kernel: mdcmd (50): spinup 3
Jul  7 15:59:38 MediaTower kernel: mdcmd (51): spinup 4
Jul  7 15:59:38 MediaTower root: 
Jul  7 15:59:38 MediaTower root: /dev/sdi:
Jul  7 15:59:38 MediaTower root:  setting standby to 0 (off)
Jul  7 15:59:42 MediaTower emhttpd: Stopping services...
Jul  7 15:59:42 MediaTower emhttpd: shcmd (60465): /etc/rc.d/rc.libvirt stop
Jul  7 16:00:42 MediaTower root: Domain 992b2eed-7034-08a5-5624-8a8ad130efd9 is being shutdown

[JonathanM]

Sounds like the batteries in the UPS are failing.

[a12vman]

I am ordering a new set of batteries. I will do a plugs-out test while the array is offline once I get the new batteries and they are fully charged.

[civic95man]

or the batteries need to be calibrated

[Jerky_san]

7 minutes ago, civic95man said:

or the batteries need to be calibrated

Yeah I was thinking they should plug the battery USB cable into a machine that is running the manufactures software and run a self test on it. See what it says. Usually the self test calibrates the battery like your saying.

[JonathanM]

6 hours ago, a12vman said:

I will do a plugs-out test

Don't actually unplug it, switch off the power to the outlet. Removing the plug breaks the ground circuit, and has the possibility of damaging equipment as any current imbalance travels through network or other still connected cables.

 

You can do a full live test without hurting anything, but it's a little complicated. You need 2 power strips, plug the UPS into one, and the stuff that's normally plugged into the UPS into the other. Approximate the load as accurately as possible, and load up the UPS output to match. Incandescent light bulbs make good small loads, space heaters for stuff above 250ish watts.

 

Run the server up to typical usage, all containers and VM's running as normal, then turn off the power strip running the UPS. The UPS should signal the outage, and hopefully the server will fully shut down before the UPS quits powering the dummy load. Ideally it should be totally done before the batteries reach 50% charge, so as not to stress them.

[a12vman]

I powered down the server and installed 3 new 12.7ah batteries into the UPS.

 

Powered up the UPS and ran a self-test/diagnostic.  Powered my server back up.  

 

Brought the server back to normal load with dockers/VM's running.

 

Shut off the breaker to the UPS. Again same issue - Server powers down within a minute of turning off the breaker.

 

If I understand the config it should keep running on batteries until it drops to 60%.

 

What is the issue?

 

 

[JonathanM]

Try filling in the blank time fields with good values.

[a12vman]

Entered legit values for Runtime Left and Time On Battery.  Cut the power at the breaker, this time no immediate shutdown.

 

I let the server run for 10 minutes on battery power then flipped the breaker back on.  

 

I now have a new set of batteries which I probably needed anyways so I think we can consider this resolved.

 

Thanks for everyone's help!

Edited July 18, 2020 by a12vman