Hi,

 

I need your help with something really strange. After years of stable operation the server did shutdown unexpectedly on its own recently. I could no find any obvious trigger in the log. This is the section of the log file where the shutdown begins (after line 2, to my best knowledge):

 

May 25 05:59:37 SS emhttp: shcmd (225): /usr/sbin/hdparm -y /dev/sdb &> /dev/null

May 25 06:26:53 SS emhttp: shcmd (226): beep -r 2

May 25 06:26:54 SS emhttp: shcmd (227): /usr/local/sbin/emhttp_event stopping_svcs

May 25 06:26:54 SS emhttp_event: stopping_svcs

May 25 06:26:54 SS kernel: mdcmd (122): nocheck

May 25 06:26:54 SS kernel: md: nocheck_array: check not active

May 25 06:26:54 SS emhttp: Stop AVAHI...

May 25 06:26:54 SS emhttp: shcmd (228): /etc/rc.d/rc.avahidaemon stop |& logger

May 25 06:26:54 SS logger: Stopping Avahi mDNS/DNS-SD Daemon: stopped

 

The shutdown was completed by

 

May 25 06:27:28 SS rc.unRAID[28349][28429]: Array Stopped

 

I am attaching the full log file.

 

The server is connected to an UPS but there was no power outage. I also did not receive any hard drive over-temp shutdown warnings.

 

The only thing that has happened compared to before was that installed a few more fans in the case a few days before.

 

This has not re-occurred since but I am nervous as I am about to replace my parity drive and don't want this to happen during the process.

 

I am on 5.0.6. Does anybody have a hunch what is going on here?

 

Thanks so much for your help!

 

Thomas

Log_1_-_syslog-20150525-062728.zip

Did you receive e-mail messages from the server? It appears to be sending.

The syslog shows a very orderly shutdown, at 06:26:54, beginning with a stop of the array, and continuing with no delay to stop all services and plugins, then Linux itself.  The fact that there is no time between array stop and shutdown means it couldn't have been the Stop button accidentally clicked, so that only leaves Powerdown being called somehow.  Powerdown is the only process I know of that conducts a complete shutdown (safe and orderly) even if array is started.  The only ways I know to initiate Powerdown are (1) run it at a command prompt, (2) set it up with a cron entry for a scheduled shutdown, (3) start it from the UPS software because of power outage, (4) press the power button on the server, and (5) press the Ctl-Alt-Del key combo on the attached keyboard.

 

There's one other oddity, a double beep occurs (and is logged!) just before the orderly shutdown.  Perhaps it's Powerdown doing it, but I've never heard it on my system, and it doesn't make any sense to log the beep command!  So if it's not Powerdown, then it's something you should be very interested in, partly because it's a strange thing to log, and because it occurs immediately before Powerdown appears to be called!  I'd check your plugins and scripts.

There's one other oddity, a double beep occurs (and is logged!) just before the orderly shutdown.  Perhaps it's Powerdown doing it, but I've never heard it on my system, and it doesn't make any sense to log the beep command!  So if it's not Powerdown, then it's something you should be very interested in, partly because it's a strange thing to log, and because it occurs immediately before Powerdown appears to be called!  I'd check your plugins and scripts.

At least throughout the 6 series, I have always had a double beep when the system goes for a shutdown or reboot when I initiate it by the GUI.  No idea if powerdown is doing that since I always have it installed.  And, just checked my last log, and that beep is logged also.

 

EDIT:  Just uninstalled powerdown, and the double beep still happens.

Just checked a log powerdown saved and I have the beep also. And I think I have always had it on reboot, which is remarkable because I'm barely capable of hearing it.

@dgaschk: the last mail sent 1.5 hours before the shutdown was a disk temperature warning for my Seagate cache disk reaching at 46C. This happens regularly. Nothing special. There were no emergency temp shutdown warnings. I get them when I e.g. rebuild parity and some drives hit 50C. I never reached an automatic shutdown temperature. At the time of the shutdown the server was idle, everybody was sleeping and the drive should have been spun-down.

 

@RobJ: 1,2 3 and 5 don't seem to apply. No recent changes to the system. Everybody was sleeping. Re. 3. I checked my UPS and no power outage was logged there. Leaves 4. I checked with my kids and nobody claims to have done anything. The buttons are also not too easy to press.

 

A command line "shutdown now" start with:

May 29 10:00:41 SS shutdown[20176]: shutting down for system reboot

May 29 10:00:43 SS init: Switching to runlevel: 6

May 29 10:00:50 SS rc.unRAID[20204][20205]: Powerdown V2.06

 

A press of the power button with:

May 29 10:06:49 SS powerdown[10650]: Powerdown initiated

May 29 10:06:49 SS powerdown[10654]: Powerdown V2.06

May 29 10:06:49 SS rc.unRAID[10656][10657]: Processing /etc/rc.d/rc.unRAID.d/ kill scripts.

 

No beep.

 

I am getting the dual beep with a normal GUI shutdown but then the logging starts with:

May 29 10:15:33 SS emhttp: shcmd (69): /usr/local/sbin/emhttp_event stopping_svcs

May 29 10:15:33 SS emhttp_event: stopping_svcs

May 29 10:15:33 SS emhttp: Stop AVAHI...

May 29 10:15:33 SS emhttp: shcmd (70): /etc/rc.d/rc.avahidaemon stop |& logger

...

May 29 10:15:49 SS emhttp: shcmd (124): beep -r 2

May 29 10:15:50 SS emhttp: mdcmd: write: Invalid argument

May 29 10:15:50 SS emhttp: Stop AVAHI...

May 29 10:15:50 SS emhttp: shcmd (125): /etc/rc.d/rc.avahidaemon stop |& logger

 

Still different from the log with the automatic shutdown

@RobJ: 1,2 3 and 5 don't seem to apply. No recent changes to the system. Everybody was sleeping. Re. 3. I checked my UPS and no power outage was logged there. Leaves 4. I checked with my kids and nobody claims to have done anything. The buttons are also not too easy to press.

 

Any chance you have a cat? 

@RobJ: 1,2 3 and 5 don't seem to apply. No recent changes to the system. Everybody was sleeping. Re. 3. I checked my UPS and no power outage was logged there. Leaves 4. I checked with my kids and nobody claims to have done anything. The buttons are also not too easy to press.

 

Any chance you have a cat? 

Are you referring to this?

I always suspected that cats got their name from cat-astrophe but none has been spotted inside here. But then you never know, they like to be out at night.

 

 

@RobJ: 1,2 3 and 5 don't seem to apply. No recent changes to the system. Everybody was sleeping. Re. 3. I checked my UPS and no power outage was logged there. Leaves 4. I checked with my kids and nobody claims to have done anything. The buttons are also not too easy to press.

 

Any chance you have a cat? 

Are you referring to this?

 

I hadn't seen that thread, but it really doesn't surprise me.  I've heard of other stories of cat behavior, with unexpected results, easy to blame by some on gremlins!  And there's that flickering drive activity light, often near the power button, tempting young and playful cats.

The BIOS log also does not reveal anything at the time of the boot. But I have a few single ECC failures. Not sure what the board does if there is a dual one.