VelcroBP Posted July 26, 2012 Share Posted July 26, 2012 Recently had a power outage. I have APCUPSD installed and the unRaid server is the only device plugged into my APC BE550G. When I came home, I saw the lights still on and the fans spinning, so I thought "all is well". However, I was unable to access either webUI (standard unRAID or unMENU), or get a connection via telnet. I went to the tower and did a short-press of the power button hoping to kick of the Clean Powerdown: No Dice. I then did a long-press hard shutdown. Rebooted, and sure enough a parity check began. However, it only lasted a few minutes (I have 3 2tb drives w/ ~2.5tb of data). I then verified that the APCUPSD and Clean Powerdown were set to re-install. I wanted to unmount the array to pull the plug on the UPS in order to test (as of this point I had a monitor and keyboard attached at the server). However, after clicking "unmount array" in unMenu, I immediately lost connection again from the web UI and telnet. I went to the terminal, and was able to log in and run ETHTOOL, so I figured I"d just restart the array and try again, however the short-press would not respond, nor would "Powerdown" or "Shutdown". Another hard-reboot. Am now in the middle of another parity check (this one appears to be taking the normal time i.e. FOREVER!!). What might gone wrong with the initial APCUPSD-initiated clean-powerdown during the storm earlier? Is unmounting the array from unMenu normally a bad idea? My server has been running smoothly since I built it a few months back, so this is my first real instance of having to troubleshoot. I"m a total Linux neophyte, so I'm looking for any advice on how I may have borked my configurations or what I might have done wrong. Thanks. Link to comment
Joe L. Posted July 27, 2012 Share Posted July 27, 2012 Recently had a power outage. I have APCUPSD installed and the unRaid server is the only device plugged into my APC BE550G. When I came home, I saw the lights still on and the fans spinning, so I thought "all is well". However, I was unable to access either webUI (standard unRAID or unMENU), or get a connection via telnet. I went to the tower and did a short-press of the power button hoping to kick of the Clean Powerdown: No Dice. I then did a long-press hard shutdown. Rebooted, and sure enough a parity check began. However, it only lasted a few minutes (I have 3 2tb drives w/ ~2.5tb of data). I then verified that the APCUPSD and Clean Powerdown were set to re-install. I wanted to unmount the array to pull the plug on the UPS in order to test (as of this point I had a monitor and keyboard attached at the server). However, after clicking "unmount array" in unMenu, I immediately lost connection again from the web UI and telnet. I went to the terminal, and was able to log in and run ETHTOOL, so I figured I"d just restart the array and try again, however the short-press would not respond, nor would "Powerdown" or "Shutdown". Another hard-reboot. Am now in the middle of another parity check (this one appears to be taking the normal time i.e. FOREVER!!). What might gone wrong with the initial APCUPSD-initiated clean-powerdown during the storm earlier? Is unmounting the array from unMenu normally a bad idea? My server has been running smoothly since I built it a few months back, so this is my first real instance of having to troubleshoot. I"m a total Linux neophyte, so I'm looking for any advice on how I may have borked my configurations or what I might have done wrong. Thanks. Did you install the clean-powerdown IN ADDITION to the apcupsd package? (it is needed to perform the actual powerdown) You do not need to un-mount the array from unMENU. (normally not a good idea to use it unless the unMENU screen is not available or is unable to un-mount disks because you have processes keeping them busy.) Joe L. Link to comment
VelcroBP Posted July 27, 2012 Author Share Posted July 27, 2012 I do have Clean Powerdown installed and set to reinstall upon reboot as well. As for un-mounting, I actually meant "stop" the array, I should have proofread my post first. What is the proper procedure for testing that Apcupsd and Clean Powerdown are configured properly? Link to comment
Joe L. Posted July 27, 2012 Share Posted July 27, 2012 I do have Clean Powerdown installed and set to reinstall upon reboot as well. As for un-mounting, I actually meant "stop" the array, I should have proofread my post first. What is the proper procedure for testing that Apcupsd and Clean Powerdown are configured properly? Easy. For the test Plug your server DIRECTLY into the AC outlet. (bypassing the UPS) Plug a 100 watt lamp into the UPS. (it will substitute as a load for the UPS.) Watch the messages in the syslog by typing tail -f /var/log/syslog Now, you can simulate a power outage by un-plugging the UPS from the wall. (to make it easier, you can plug the UPS into a power-strip, andthen use the switch on the power-strip to cut the power from the wall outlet to the UPS. The UPS will power the lamp and should stop the array before turning off the lamp. You will see the messages in the syslog. you may find that add-on programs are keeping the power-down-script from stopping the array. Joe L. Link to comment
jowi Posted July 27, 2012 Share Posted July 27, 2012 For the test Plug your server DIRECTLY into the AC outlet. (bypassing the UPS) Plug a 100 watt lamp into the UPS. (it will substitute as a load for the UPS.) Now, you can simulate a power outage by un-plugging the UPS from the wall. The UPS will power the lamp and should stop the array before turning off the lamp. Don't forget that even though the server is not plugged in to the UPS for this test, the USB cable should still be connected from the UPS to the server, so that the ups can still signal the server. Link to comment
Joe L. Posted July 27, 2012 Share Posted July 27, 2012 For the test Plug your server DIRECTLY into the AC outlet. (bypassing the UPS) Plug a 100 watt lamp into the UPS. (it will substitute as a load for the UPS.) Now, you can simulate a power outage by un-plugging the UPS from the wall. The UPS will power the lamp and should stop the array before turning off the lamp. Don't forget that even though the server is not plugged in to the UPS for this test, the USB cable should still be connected from the UPS to the server, so that the ups can still signal the server. I did not explicitly say that, but yes, that USB cable must be in place. Link to comment
VelcroBP Posted July 27, 2012 Author Share Posted July 27, 2012 Alright, so,.. apologies for wasting time here with my foolishness... but have a laugh at my expense! Turns out that my UPS only provides battery backup to HALF of the outlets! I need to learn to not freak out and look at the most rudimentary steps first when these things happen! On the plus side, I very much appreciate Joe L.'s test methodology. I now am confident that APCUPSD is properly shutting down the array (provided I plug it in to the proper outlets!). On a side note, I did observe that the UPS is not shut off after the server powers down (the lamp remained on). No worries, as all I am ultimately concerned about is a protected array. Thanks for your help, yet again, unRaid community! Link to comment
JonathanM Posted July 28, 2012 Share Posted July 28, 2012 On a side note, I did observe that the UPS is not shut off after the server powers down (the lamp remained on). That means you MUST configure the server to stay off until you manually push the power button, if it wasn't already set that way. Typically the corresponding resume on power cmos entry will say something about power on, previous state, or off. It must be set to off. Reason being, if the power fails and the battery backup runs totally down, when the power resumes, the server will be starting back up with the batteries almost totally dead. If the power blinks before the batteries are recharged, it will most likely just die on you. Link to comment
JonathanM Posted July 28, 2012 Share Posted July 28, 2012 Is this a configuration setting in APCUPSD? Nope. That's a setting that is controlled by the motherboard's BIOS, and each motherboard model is a little different. Check your motherboard manual, or poke around in the cmos settings. That setting controls what happens when you apply power to the back of the tower. It can automatically power on, stay off, or resume the power state when the power was cut last. When it's set to stay off, you have to press the power button to start, which is what you want so the server doesn't automatically power up when you aren't there to monitor the process. Link to comment
jowi Posted July 28, 2012 Share Posted July 28, 2012 Why is it neccesary to shut down the ups as well? Doesnt it just stop when it runs out of battery power? Offcourse it should have shutdown the server before that. But whats the problem? Link to comment
JonathanM Posted July 28, 2012 Share Posted July 28, 2012 Why is it neccesary to shut down the ups as well? Doesnt it just stop when it runs out of battery power? Offcourse it should have shutdown the server before that. But whats the problem? Not necessary, just very desirable. If you can save the UPS batteries from fully discharging, it keeps the recharge time lower, and helps preserve the overall life of the batteries. Even deep cycle batteries don't particularly like being completely drained. Plus, a well designed UPS ($$$) can get you back up and running without direct user intervention. It flows like this... Mains power loss - backup signals server estimated runtime remaining Server decides to shut down - signals backup to cut power Mains power restored - backup recharges batteries to a healthy reserve level then switches on power to server Server automatically boots when power is applied. If your UPS doesn't work this way, you want the server to stay off until you intervene, even if the power comes back. Most cheap UPS automatically turn the power to the controlled outlets back on immediately on mains restoration, even if the batteries are completely dead, which sets you up for a server crash if the power flickers before the batteries charge. Link to comment
Joe L. Posted July 30, 2012 Share Posted July 30, 2012 Very clear now, thanks! The other scenerio is Power is lost for X minutes APCUPSD stops the array and powers down the server. (it is set to start when power returns) the UPS continues to power the server... but power is restored. The UPS has no way to re-start the server. The server is powered down. It never stopped supplying power to the server. There is a setting in the APCUPSD config to power down the UPS after powering down the server. This might be a minute or two AFTER the server is powered down, as the UPS wants to make sure the server had time to power down cleanly. If you have that setting in to power down the UPS set to "YES" then one of the unRAID shutdown scripts (/etc/rc.d/rc.6) is modified so the last thing it does is send the message to the UPS the server is shutting down. The UPS then knows it can shut itself down soon thereafter. The line near the end of /etc/rc.d/rc.6 that originally invokes the poweroff command: /sbin/poweroff is changed to send a message to the UPS first to have it shut down, then power-off: /etc/apcupsd/apccontrol killpower; /sbin/poweroff Joe L. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.