Jump to content
dlandon

Dealing with unclean shutdowns

57 posts in this topic Last Reply

Recommended Posts

There are several things you need to check in your Unraid setup to help prevent the dreaded unclean shutdown.  There are several timers that you need to adjust for your specific needs.

 

There is a timer in the Settings->VM Manager->VM Shutdown time-out that needs to be set to a high enough value to allow your VMs time to completely shutdown.  Switch to the Advanced View to see the timer.  Windows 10 VMs will sometimes have an update that requires a shutdown to perform.  These can take quite a while and the default setting of 60 seconds in the VM Manager is not long enough.  If the VM Manager timer setting is exceeded on a shutdown, your VMs will be forced to shutdown.  This is just like pulling the plug on a PC.  I recommend setting this value to 300 seconds (5 minutes) in order to insure your Windows 10 VMs have time to completely shutdown.

 

The other timer used for shutdowns is in the Settings->Disk Settings->Shutdown time-out.  This is the overall shutdown timer and when this timer is exceeded, an unclean shutdown will occur.  This timer has to be more than the VM shutdown timer.  I recommend setting it to 420 seconds (7 minutes) to give the system time to completely shut down all VMs, Dockers, and plugins.

 

These timer settings do not extend the normal overall shutdown time, they just allow Unraid the time needed to do a graceful shutdown and prevent the unclean shutdown.

 

One of the most common reasons for an unclean shutdown is having a terminal session open.  Unraid will not force them to shut down, but instead waits for them to be terminated while the shutdown timer is running.  After the overall shutdown timer runs out, the server is forced to shutdown.  If you have the Tips and Tweaks plugin installed, you can specify that any bash or ssh sessions be terminated so Unraid can be gracefully shutdown and won't hang waiting for them to terminate (which they won't without human intervention).

 

If you server seems hung and nothing responds, try a quick press of the power button.  This will initiate a shutdown that will attempt a graceful shutdown of the server.  If you have to hold the power button to do a hard power off, you will get an unclean shutdown.

 

If an unclean shutdown does occur because the overall "Shutdown time-out" was exceeded, Unraid will attempt to write diagnostics to the /log/ folder on the flash drive.  When you ask for help with an unclean shutdown, post the /log/diagnostics.zip file.  There is information in the log that shows why the unclean shutdown occurred.

Edited by dlandon
  • Like 11
  • Upvote 2

Share this post


Link to post
2 hours ago, dlandon said:

If you have the Tips and Tweaks plugin installed, you can specify that any bash or ssh sessions be terminated so unRAID can be gracefully shutdown and won't hang waiting for them to terminate (which they won't without human intervention).

 

Am I correct in assuming that one should enter    ssh  bash   in the field for the   Processes to kill before Array is Stopped:  parameter?

 

EDIT:

3 minutes ago, dlandon said:

Separate the processes with a comma.  Hover over the "Processes to kill before the Array is Stopped" and then click when the ? appears and you will see the Help.

 

So per your advice, like this   ssh,bash     

Edited by Frank1940

Share this post


Link to post
16 minutes ago, Frank1940 said:

This is should be a Sticky!!!!

I'll ask a moderator to sticky it.

Share this post


Link to post
5 minutes ago, Frank1940 said:

 

Am I correct in assuming that one should enter    ssh  bash   in the field for the   Processes to kill before Array is Stopped:  parameter?

Separate the processes with a comma.  Hover over the "Processes to kill before the Array is Stopped" and then click when the ? appears and you will see the Help.

Share this post


Link to post
2 hours ago, Frank1940 said:

 

In the original post, I would suggest revising this suggested action (and the one for the VM's time out) to include the name of actual parameter to be modified.  Perhaps like this:

 

             "The other timer used for shutdowns (called Shutdown time-out:)  is in the Settings->Disk Settings."

 

I always appreciate when there are a large number of parameters in a location that the one requiring action be explicitly named.  It can save a lot of confusion (and an occasional stupid mistake on my part). 

Changed.  Thanks for the suggestion.

Share this post


Link to post
8 hours ago, Frank1940 said:

So per your advice, like this   ssh,bash 

Yes.

Share this post


Link to post
15 hours ago, Darksurf said:

OMG thank you!

Can I assume you found this information helpful?

Share this post


Link to post
Can I assume you found this information helpful?
I definitely did! Thanks!

Sent from my ONEPLUS A3000 using Tapatalk

Share this post


Link to post
Can I assume you found this information helpful?

I also found it v.useful ... but i have a question regarding the disk settings timeout. I changed mine to 420 as recommended but on a 'clean' powerdown instigated by myself the system just shut down as normal within what seemed to be the default 60secs. Is this right or should i expect every shutdown to last 420 secs minimum? Or, as seems to be the case once the disks are 'safely' unmounted the system just powers down overriding the timer setting? I'm testing an APC ups and having all sorts of probs getting the values right.

 

Sent from my LG-D855 using Tapatalk

 

 

 

Share this post


Link to post
2 hours ago, superloopy1 said:

I also found it v.useful ... but i have a question regarding the disk settings timeout. I changed mine to 420 as recommended but on a 'clean' powerdown instigated by myself the system just shut down as normal within what seemed to be the default 60secs. Is this right or should i expect every shutdown to last 420 secs minimum? Or, as seems to be the case once the disks are 'safely' unmounted the system just powers down overriding the timer setting? I'm testing an APC ups and having all sorts of probs getting the values right.

 

Sent from my LG-D855 using Tapatalk

 

 

 

From first post:

On 3/5/2018 at 8:57 AM, dlandon said:

These timer settings do not extend the normal overall shutdown time, they just allow unRAID the time needed to do a graceful shutdown and prevent the unclean shutdown.

The 420 seconds is the maximum time to allow unRAID to cleanly shutdown before forcing a shutdown.  If the clean shutdown occurs in 60 seconds, then the system will shutdown in 60 seconds.

Share this post


Link to post
The 420 seconds is the maximum time to allow unRAID to cleanly shutdown before forcing a shutdown.  If the clean shutdown occurs in 60 seconds, then the system will shutdown in 60 seconds.

Ok ... thanks for that but i'm still not fully understanding. I'll have another go. Its the relationship between vm and disk timeouts thats got me. I had a vm up and running and the system shut down just as before. If that vm had overrun to lets say 120 secs the system would have 'waited' on the vm irrespective of other timers being met, yes?  

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

Share this post


Link to post

The is great advise @dlandon as I had several unclean shutdown's in the past weeks (when I tried to shutdown either via Main/Array Operation/PowerDown or via Dynamix System Buttons).

 

I followed your advise and have modified these settings:

  • Settings->VM Manager->VM Shutdown time-out 300 seconds
  • Settings->Disk Settings->Shutdown time-out 420 seconds
  • Tips and Tweaks plugin:  Processes to kill before Array is Stopped: ssh, bash

Here is the diagnostics from yesterday....I can't see that anything from the 3 new settings were preventing my array to shutdown but someone smarter than me might identify the root cause.

 

Thanks a lot.

 

Edited by EdgarWallace

Share this post


Link to post
On 3/27/2018 at 9:25 AM, superloopy1 said:

Ok ... thanks for that but i'm still not fully understanding. I'll have another go. Its the relationship between vm and disk timeouts thats got me. I had a vm up and running and the system shut down just as before. If that vm had overrun to lets say 120 secs the system would have 'waited' on the vm irrespective of other timers being met, yes?  

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

No.  You have two timers running at the same time.  The VM shutdown timer needs to be shorter than the shutdown timer so the VMs have a chance to time out.  If the VM timer runs out, the VMs are forced to shut down.

 

VM timer              =======================>  VMs are forced to shut down if the timer runs out

Shutdown Timer ========================================================> Unraid is forced to shutdown if the timer runs out

Edited by dlandon

Share this post


Link to post
9 hours ago, EdgarWallace said:

The is great advise @dlandon as I had several unclean shutdown's in the past weeks (when I tried to shutdown either via Main/Array Operation/PowerDown or via Dynamix System Buttons).

 

I followed your advise and have modified these settings:

  • Settings->VM Manager->VM Shutdown time-out 300 seconds
  • Settings->Disk Settings->Shutdown time-out 420 seconds
  • Tips and Tweaks plugin:  Processes to kill before Array is Stopped: ssh, bash

Here is the diagnostics from yesterday....I can't see that anything from the 3 new settings were preventing my array to shutdown but someone smarter than me might identify the root cause.

 

Thanks a lot.

tower-diagnostics-20180423-2310.zip

Your shutdown occurred within a few minutes.  It doesn't seem like the timers worked at all.  Double check the settings.

 

Your shutdown timer is set to 60 seconds.

Your VM timer is set to 60 seconds.

Share this post


Link to post

Perhaps a stupid question, but how can I see in the unraid-diagnostics file if there was an 'unclean' shutdown?

This morning the ups 'nut' client initiated a shutdown 5min after being on battery as configured.

 

May  9 04:09:06 unraid upsmon[5804]: UPS xxx@removed.com on battery
May  9 04:14:10 unraid upsmon[5804]: Signal 10: User requested FSD
May  9 04:14:10 unraid upsmon[5804]: Executing automatic power-fail shutdown
May  9 04:14:10 unraid upsmon[5804]: Auto logout and shutdown proceeding
May  9 04:14:15 unraid shutdown[1509]: shutting down for system halt
May  9 04:14:15 unraid init: Switching to runlevel: 0

 

The very last entry in the syslog.txt is

 

May  9 04:15:48 unraid root: Generating diagnostics...

 

So what does this mean exactly?

 

In the meantime I also changed the timers as suggested.

Share this post


Link to post
6 hours ago, themaxxz said:

Perhaps a stupid question, but how can I see in the unraid-diagnostics file if there was an 'unclean' shutdown?

This morning the ups 'nut' client initiated a shutdown 5min after being on battery as configured.

 

May  9 04:09:06 unraid upsmon[5804]: UPS xxx@removed.com on battery
May  9 04:14:10 unraid upsmon[5804]: Signal 10: User requested FSD
May  9 04:14:10 unraid upsmon[5804]: Executing automatic power-fail shutdown
May  9 04:14:10 unraid upsmon[5804]: Auto logout and shutdown proceeding
May  9 04:14:15 unraid shutdown[1509]: shutting down for system halt
May  9 04:14:15 unraid init: Switching to runlevel: 0

 

The very last entry in the syslog.txt is

 

May  9 04:15:48 unraid root: Generating diagnostics...

 

So what does this mean exactly?

 

In the meantime I also changed the timers as suggested.

Look on the flash drive in the /logs/ folder.

Share this post


Link to post
13 hours ago, dlandon said:

Look on the flash drive in the /logs/ folder.

Thanks, indeed that is where I found the diagnostic file with the syslog.txt file from which I quoted the previous logging.

But it's not clear to me how a clean or unclean shutdown logging should look like.

Share this post


Link to post
11 minutes ago, themaxxz said:

Thanks, indeed that is where I found the diagnostic file with the syslog.txt file from which I quoted the previous logging.

But it's not clear to me how a clean or unclean shutdown logging should look like.

 

If the unclean shutdown is because power is lost then of course it has no opportunity to log anything.

 

Not sure what happens if an attempted shutdown times out. Maybe that would be logged but in that case I think it would say something in the logs about it.

Share this post


Link to post
20 hours ago, themaxxz said:

May  9 04:15:48 unraid root: Generating diagnostics...

A clean shutdown wouldn't create diags, they are only created unRAID coudn't do a clean shutdown after the set timeout.

Share this post


Link to post

@themaxxz, could your UPS battery have "completely" discharged during this attempted shutdown?  This would result in the UPS shutting off as soon as it detected that battery state.  What are your conditions for a initiation of the server's shutdown?    (My condition is 30 seconds on the battery!)    

Share this post


Link to post
2 hours ago, johnnie.black said:

A clean shutdown wouldn't create diags, they are only created unRAID coudn't do a clean shutdown after the set timeout.

Ok, so it was an unclean shutdown in this case, and most likely due to having a too short timeout set.

Thanks.

Edited by themaxxz

Share this post


Link to post
2 hours ago, Frank1940 said:

@themaxxz, could your UPS battery have "completely" discharged during this attempted shutdown?  This would result in the UPS shutting off as soon as it detected that battery state.  What are your conditions for a initiation of the server's shutdown?    (My condition is 30 seconds on the battery!)    

No the UPS never ran out of power.  My condition is 5min on battery.  The rest of the network stayed up and was still up when power returned after 30min or so.

The server just completed a parity check (which I initiated) and everything checks out.

I just didn't know about the shutdown time-out settings waiting for a graceful shutdown.

 

 

Edited by themaxxz

Share this post


Link to post

I'm wanting a clean reboot on a daily basis and I don't want to cause unclean shutdowns.  I run plex, mysql, pydio and other dockers.  Will this script work to cleanly shutdown the dockers, then reboot?  I'd put this in the User Scripts plugin...  

 

/etc/rc.d/rc.docker stop && /etc/rc.d/rc.docker start
/usr/local/sbin/powerdown -r

Thanks in advance!

Edited by jeffreywhunter

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.