Dealing with unclean shutdowns

[JorgeB]

14 hours ago, anthem221 said:

I had a lot of log entry's that something is wrong, I fixed everything I could but the original failure still remains. 

One of the call traces is kvm related, does it still crash with VMs off?

[anthem221]

I have two VMs running, one windows 10 I had that of to test, Still crashes. The other one....I can't really turn of because that's a pfsense...if I turn that off I can't even turn on the light in my flat 😂😂

 

But for test I can install the old router back for some days - do you think it's the pfsense? It's just running in the background with nothing attached except one pcie ethernet card.

Edited January 27 by anthem221

[elnevera]

Hi all,

 

My server has been having a few unclean shutdowns. 

 

 

As you can see from the above syslog file, things stopped on the 20th and started when I powered it up this morning. I had setup Syslog server to try and capture what was going on but got nothing. Does this point to hardware failure?

 

Thanks

[trurl]

3 hours ago, elnevera said:

setup Syslog server to try and capture what was going on but got nothing.

Do you mean syslog server didn't work for you? Or do you mean there was nothing in the resulting syslog that seemed to explain your problem?

[elnevera]

49 minutes ago, trurl said:

Do you mean syslog server didn't work for you? Or do you mean there was nothing in the resulting syslog that seemed to explain your problem?

Nothing notable in the Syslog, The last entry in Syslog was 15:32 on the 20th but I know that the server was still active until 03:39 on the 21st as my Scrypted container was still sending video to HomeKit until that point. 

[deerow]

Hello,

 

I had a few unclean shutdowns in the past couple days. I am unsure what the cause of it, but I believe it due to me pre-clearing too many drives at once?

I just had just upgraded my server to 32gb of ram and the whole array(partiy/data) been replaced with brand new drives last week. I am preclearing the old drives and getting ready to toss them since they have a number of attribute failures. Since all the changes my server seems to be unstable.

Anyone able to find out why this is happening, and double check nothing alarming in my logs?

I see these errors in the syslog.txt but unsure if it is the cause of it

diagnostics-20240401-1022.zip

Edited April 1 by deerow

[FrozenGamer]

I have had a decent number of unclean shutdowns for maybe the last 8 months.. (every power outtage even with UPS) All have taken place when power went out.. in the last instance, the power went out, and when my server was restarted it flagged as unclean and started the parity check.. I changed a few UPS settings, assuming it was this causing it.. sometime during the parity check the machine powered down and i don't know why.. Perhaps because i messed with the settings for ups shutdown... I do not believe there was an additional power outage because another computer was booted when i got home (i am gone a lot for work, long periods of time)

As far as i can tell on the flash drive under logs - their is only a diagnostics which i created myself after i booted server again.  

I had my hard drive timeout set to 60, just changed it to 420

I do not have any VM's

I did a test of the stop array command and it was maybe 30 seconds.. (but this is fresh boot of server if that matters)

I put ssh,bash settings in case it was due to a terminal session being set.. which is a possibility.

I am attaching my latest diagnostics, even though as far as i can tell it doesn't indicate shutdown success/failure.. But maybe it has important info.

I am also attaching a screenshot of my latest ups shutdown settings for comments if something should be changed.

Is there something i can do to help ensure that the syslog records the shutdown process during a power outage?

 

Hopefully i didn't miss anything in the thread when i read through it.

Thanks

 

tower-diagnostics-20240427-1416.zip

Edited April 27 by FrozenGamer

[JorgeB]

10 hours ago, FrozenGamer said:

As far as i can tell on the flash drive under logs - their is only a diagnostics which i created myself after i booted server again.  

If Unraid forces the shutdown there should be diags there, but you can try enabling the syslog server, and post that after the next unclean one.

[tgiannak]

Hello, 

 

after 3 months with stable server, the last 2 days started to make reboots!! and parity sync! can someone help me with the logs? im frustrated! i didnt even update the last 3 months...

 

 

syslog-192.168.4.200.log diagnostics-20240429-1032.zip

[JorgeB]

8 minutes ago, tgiannak said:

after 3 months with stable server, the last 2 days started to make reboots!!

If you mean the server is rebooting on its own, that's almost always a hardware problem.

[tgiannak]

5 minutes ago, JorgeB said:

If you mean the server is rebooting on its own, that's almost always a hardware problem.

can you point me to a direction to look based on the logs??

 

thanks Jorge

[JorgeB]

If it's a hardware issue most likely there won't be anything logged, you can start by running memtest or using for example a different PSU.

[FrozenGamer]

Power went out or flickered again sometime while i was gone and server was off again.. I am starting to wonder if the power supply itself isn't able to keep the load of the entire server and 32 drives without powering down.. I say this because i use a similar costco UPS on my desktop and i have been coming home to see computer powered off a few times and ups down.. (not unraid server)

 

this has never happened on unraid server as far as i know its been power outtages.  

I think desktop is lower power load, i just eliminated the ups from desktop..

If i was to choose to do a test by unplugging my unraid server ups from wall and seeing if it supports the load, i assume it would be best to do so when the server array was down and drives spinning still?  So as not to damage file integrity?  Any other ideas on how to test etc.  

[FrozenGamer]

I can't promise it is not a hardware issue (as in memory etc), I really dont think it is.. but it may be a ups issue?

[JorgeB]

8 hours ago, FrozenGamer said:

but it may be a ups issue?

It could be.

[FrozenGamer]

I have determined this is a 6 year old ups, did the simple unplug of wall to see how long it would run with just the 2 hard drive caddies and the ups shut off in 10 or so seconds, replaced it with a new ups (99 dollar CST135UC2 cyberpower from costco) and unplugged drive caddies only, says 23 minutes with that load.. 

So now that i know the ups isn't working right, i am hoping the new one will shut itself down after power down of unraid (but the 2 drive caddies cannot be shut off that i know of.)  I had read that the old similar cyberpower ups was not cable of shutting itself off afterwards.. 

 

I assume this maybe degraded the ups? or just the 6 years of use?

 

  Looking into how to recondition the batteries, but i assume i should just buy a few more from costco and not worry about it, maybe even a separate one for the caddies (32 drives) and the box amd 5900x without a graphics card)

 

Thanks everyone for all the assistance.. unraid is pretty amazing and the support in the forums is above and beyond.

 

Edit- partial answers to my own questions.. unplug worked fine and shut down the desktop with unraid.. It also triggered the auto shutdown of the UPS which set to shut down in 59 minutes.. This will not work with 32 drives connnected to it.. Looking into how to change that shutdown time with https://www.cyberpowersystems.com/product/software/power-panel-business/powerpanel-business-windows/

Edited June 4 by FrozenGamer
more info. notes.

[FrozenGamer]

Can anyone let me know if these settings look good? I am going to unplug test it now.