Dealing with unclean shutdowns

[dlandon]

There are several things you need to check in your Unraid setup to help prevent the dreaded unclean shutdown.  There are several timers that you need to adjust for your specific needs.

 

There is a timer in the Settings->VM Manager->VM Shutdown time-out that needs to be set to a high enough value to allow your VMs time to completely shutdown.  Switch to the Advanced View to see the timer.  Windows 10 VMs will sometimes have an update that requires a shutdown to perform.  These can take quite a while and the default setting of 60 seconds in the VM Manager is not long enough.  If the VM Manager timer setting is exceeded on a shutdown, your VMs will be forced to shutdown.  This is just like pulling the plug on a PC.  I recommend setting this value to 300 seconds (5 minutes) in order to insure your Windows 10 VMs have time to completely shutdown.

 

The other timer used for shutdowns is in the Settings->Disk Settings->Shutdown time-out.  This is the overall shutdown timer and when this timer is exceeded, an unclean shutdown will occur.  This timer has to be more than the VM shutdown timer.  I recommend setting it to 420 seconds (7 minutes) to give the system time to completely shut down all VMs, Dockers, and plugins.  If you have remote SMB or NFS mounts in Unassigned Devices you need to account for time for them to time out if the remote server has gone off-line when unmounting.  I recommend about 45 seconds for each remote mount.  They are unmounted sequentially, so you need to account for 45 seconds for each one.

 

These timer settings do not extend the normal overall shutdown time, they just allow Unraid the time needed to do a graceful shutdown and prevent the unclean shutdown.

 

One of the most common reasons for an unclean shutdown is having a terminal session open.  Unraid will not force them to shut down, but instead waits for them to be terminated while the shutdown timer is running.  After the overall shutdown timer runs out, the server is forced to shutdown.  If you have the Tips and Tweaks plugin installed, you can specify that any bash or ssh sessions be terminated so Unraid can be gracefully shutdown and won't hang waiting for them to terminate (which they won't without human intervention).

 

If you server seems hung and nothing responds, try a quick press of the power button.  This will initiate a shutdown that will attempt a graceful shutdown of the server.  If you have to hold the power button to do a hard power off, you will get an unclean shutdown.

 

If an unclean shutdown does occur because the overall "Shutdown time-out" was exceeded, Unraid will attempt to write diagnostics to the /log/ folder on the flash drive.  When you ask for help with an unclean shutdown, post the /log/diagnostics.zip file.  There is information in the log that shows why the unclean shutdown occurred.

Edited December 29, 2019 by dlandon

[Frank1940]

2 hours ago, dlandon said:

If you have the Tips and Tweaks plugin installed, you can specify that any bash or ssh sessions be terminated so unRAID can be gracefully shutdown and won't hang waiting for them to terminate (which they won't without human intervention).

 

Am I correct in assuming that one should enter    ssh  bash   in the field for the   Processes to kill before Array is Stopped:  parameter?

 

EDIT:

3 minutes ago, dlandon said:

Separate the processes with a comma.  Hover over the "Processes to kill before the Array is Stopped" and then click when the ? appears and you will see the Help.

 

So per your advice, like this   ssh,bash     

Edited March 5, 2018 by Frank1940

[dlandon]

16 minutes ago, Frank1940 said:

This is should be a Sticky!!!!

I'll ask a moderator to sticky it.

[dlandon]

5 minutes ago, Frank1940 said:

 

Am I correct in assuming that one should enter    ssh  bash   in the field for the   Processes to kill before Array is Stopped:  parameter?

Separate the processes with a comma.  Hover over the "Processes to kill before the Array is Stopped" and then click when the ? appears and you will see the Help.

[dlandon]

2 hours ago, Frank1940 said:

 

In the original post, I would suggest revising this suggested action (and the one for the VM's time out) to include the name of actual parameter to be modified.  Perhaps like this:

 

             "The other timer used for shutdowns (called Shutdown time-out:)  is in the Settings->Disk Settings."

 

I always appreciate when there are a large number of parameters in a location that the one requiring action be explicitly named.  It can save a lot of confusion (and an occasional stupid mistake on my part). 

Changed.  Thanks for the suggestion.

[dlandon]

8 hours ago, Frank1940 said:

So per your advice, like this   ssh,bash 

Yes.

[Darksurf]

OMG thank you!

[dlandon]

15 hours ago, Darksurf said:

OMG thank you!

Can I assume you found this information helpful?

[Kewjoe]

Can I assume you found this information helpful?

I definitely did! Thanks!

Sent from my ONEPLUS A3000 using Tapatalk

[superloopy1]

Can I assume you found this information helpful?

I also found it v.useful ... but i have a question regarding the disk settings timeout. I changed mine to 420 as recommended but on a 'clean' powerdown instigated by myself the system just shut down as normal within what seemed to be the default 60secs. Is this right or should i expect every shutdown to last 420 secs minimum? Or, as seems to be the case once the disks are 'safely' unmounted the system just powers down overriding the timer setting? I'm testing an APC ups and having all sorts of probs getting the values right.

 

Sent from my LG-D855 using Tapatalk

 

 

 

[dlandon]

2 hours ago, superloopy1 said:

I also found it v.useful ... but i have a question regarding the disk settings timeout. I changed mine to 420 as recommended but on a 'clean' powerdown instigated by myself the system just shut down as normal within what seemed to be the default 60secs. Is this right or should i expect every shutdown to last 420 secs minimum? Or, as seems to be the case once the disks are 'safely' unmounted the system just powers down overriding the timer setting? I'm testing an APC ups and having all sorts of probs getting the values right.

 

Sent from my LG-D855 using Tapatalk

 

 

 

From first post:

On 3/5/2018 at 8:57 AM, dlandon said:

These timer settings do not extend the normal overall shutdown time, they just allow unRAID the time needed to do a graceful shutdown and prevent the unclean shutdown.

The 420 seconds is the maximum time to allow unRAID to cleanly shutdown before forcing a shutdown.  If the clean shutdown occurs in 60 seconds, then the system will shutdown in 60 seconds.

[superloopy1]

The 420 seconds is the maximum time to allow unRAID to cleanly shutdown before forcing a shutdown.  If the clean shutdown occurs in 60 seconds, then the system will shutdown in 60 seconds.

Ok ... thanks for that but i'm still not fully understanding. I'll have another go. Its the relationship between vm and disk timeouts thats got me. I had a vm up and running and the system shut down just as before. If that vm had overrun to lets say 120 secs the system would have 'waited' on the vm irrespective of other timers being met, yes?  

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

[EdgarWallace]

The is great advise @dlandon as I had several unclean shutdown's in the past weeks (when I tried to shutdown either via Main/Array Operation/PowerDown or via Dynamix System Buttons).

 

I followed your advise and have modified these settings:

• Settings->VM Manager->VM Shutdown time-out 300 seconds
• Settings->Disk Settings->Shutdown time-out 420 seconds
• Tips and Tweaks plugin:  Processes to kill before Array is Stopped: ssh, bash

Here is the diagnostics from yesterday....I can't see that anything from the 3 new settings were preventing my array to shutdown but someone smarter than me might identify the root cause.

 

Thanks a lot.

 

Edited April 25, 2018 by EdgarWallace

[dlandon]

On 3/27/2018 at 9:25 AM, superloopy1 said:

Ok ... thanks for that but i'm still not fully understanding. I'll have another go. Its the relationship between vm and disk timeouts thats got me. I had a vm up and running and the system shut down just as before. If that vm had overrun to lets say 120 secs the system would have 'waited' on the vm irrespective of other timers being met, yes?  

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

No.  You have two timers running at the same time.  The VM shutdown timer needs to be shorter than the shutdown timer so the VMs have a chance to time out.  If the VM timer runs out, the VMs are forced to shut down.

 

VM timer              =======================>  VMs are forced to shut down if the timer runs out

Shutdown Timer ========================================================> Unraid is forced to shutdown if the timer runs out

Edited January 6, 2019 by dlandon

[dlandon]

9 hours ago, EdgarWallace said:

The is great advise @dlandon as I had several unclean shutdown's in the past weeks (when I tried to shutdown either via Main/Array Operation/PowerDown or via Dynamix System Buttons).

 

I followed your advise and have modified these settings:

• Settings->VM Manager->VM Shutdown time-out 300 seconds
• Settings->Disk Settings->Shutdown time-out 420 seconds
• Tips and Tweaks plugin:  Processes to kill before Array is Stopped: ssh, bash

Here is the diagnostics from yesterday....I can't see that anything from the 3 new settings were preventing my array to shutdown but someone smarter than me might identify the root cause.

 

Thanks a lot.

tower-diagnostics-20180423-2310.zip

Your shutdown occurred within a few minutes.  It doesn't seem like the timers worked at all.  Double check the settings.

 

Your shutdown timer is set to 60 seconds.

Your VM timer is set to 60 seconds.

[EdgarWallace]

I corrected according to your advise and the shutdown went fine yesterday. This should be visible from the diagnostics below. Cross fingers that the issue is gone.

 

Thanks again @dlandon 

 

tower-diagnostics-20180424-2331.zip

Edited April 22, 2019 by EdgarWallace

[themaxxz]

Perhaps a stupid question, but how can I see in the unraid-diagnostics file if there was an 'unclean' shutdown?

This morning the ups 'nut' client initiated a shutdown 5min after being on battery as configured.

 

May  9 04:09:06 unraid upsmon[5804]: UPS [email protected] on battery
May  9 04:14:10 unraid upsmon[5804]: Signal 10: User requested FSD
May  9 04:14:10 unraid upsmon[5804]: Executing automatic power-fail shutdown
May  9 04:14:10 unraid upsmon[5804]: Auto logout and shutdown proceeding
May  9 04:14:15 unraid shutdown[1509]: shutting down for system halt
May  9 04:14:15 unraid init: Switching to runlevel: 0

 

The very last entry in the syslog.txt is

 

May  9 04:15:48 unraid root: Generating diagnostics...

 

So what does this mean exactly?

 

In the meantime I also changed the timers as suggested.

[dlandon]

6 hours ago, themaxxz said:

Perhaps a stupid question, but how can I see in the unraid-diagnostics file if there was an 'unclean' shutdown?

This morning the ups 'nut' client initiated a shutdown 5min after being on battery as configured.

 

May  9 04:09:06 unraid upsmon[5804]: UPS [email protected] on battery
May  9 04:14:10 unraid upsmon[5804]: Signal 10: User requested FSD
May  9 04:14:10 unraid upsmon[5804]: Executing automatic power-fail shutdown
May  9 04:14:10 unraid upsmon[5804]: Auto logout and shutdown proceeding
May  9 04:14:15 unraid shutdown[1509]: shutting down for system halt
May  9 04:14:15 unraid init: Switching to runlevel: 0

 

The very last entry in the syslog.txt is

 

May  9 04:15:48 unraid root: Generating diagnostics...

 

So what does this mean exactly?

 

In the meantime I also changed the timers as suggested.

Look on the flash drive in the /logs/ folder.

[themaxxz]

13 hours ago, dlandon said:

Look on the flash drive in the /logs/ folder.

Thanks, indeed that is where I found the diagnostic file with the syslog.txt file from which I quoted the previous logging.

But it's not clear to me how a clean or unclean shutdown logging should look like.

[trurl]

11 minutes ago, themaxxz said:

Thanks, indeed that is where I found the diagnostic file with the syslog.txt file from which I quoted the previous logging.

But it's not clear to me how a clean or unclean shutdown logging should look like.

 

If the unclean shutdown is because power is lost then of course it has no opportunity to log anything.

 

Not sure what happens if an attempted shutdown times out. Maybe that would be logged but in that case I think it would say something in the logs about it.

[JorgeB]

20 hours ago, themaxxz said:

May  9 04:15:48 unraid root: Generating diagnostics...

A clean shutdown wouldn't create diags, they are only created unRAID coudn't do a clean shutdown after the set timeout.

[Frank1940]

@themaxxz, could your UPS battery have "completely" discharged during this attempted shutdown?  This would result in the UPS shutting off as soon as it detected that battery state.  What are your conditions for a initiation of the server's shutdown?    (My condition is 30 seconds on the battery!)    

[themaxxz]

2 hours ago, johnnie.black said:

A clean shutdown wouldn't create diags, they are only created unRAID coudn't do a clean shutdown after the set timeout.

Ok, so it was an unclean shutdown in this case, and most likely due to having a too short timeout set.

Thanks.

Edited May 10, 2018 by themaxxz

[themaxxz]

2 hours ago, Frank1940 said:

@themaxxz, could your UPS battery have "completely" discharged during this attempted shutdown?  This would result in the UPS shutting off as soon as it detected that battery state.  What are your conditions for a initiation of the server's shutdown?    (My condition is 30 seconds on the battery!)    

No the UPS never ran out of power.  My condition is 5min on battery.  The rest of the network stayed up and was still up when power returned after 30min or so.

The server just completed a parity check (which I initiated) and everything checks out.

I just didn't know about the shutdown time-out settings waiting for a graceful shutdown.

 

 

Edited May 10, 2018 by themaxxz

[jeffreywhunter]

I'm wanting a clean reboot on a daily basis and I don't want to cause unclean shutdowns.  I run plex, mysql, pydio and other dockers.  Will this script work to cleanly shutdown the dockers, then reboot?  I'd put this in the User Scripts plugin...  

 

/etc/rc.d/rc.docker stop && /etc/rc.d/rc.docker start
/usr/local/sbin/powerdown -r

Thanks in advance!

Edited May 11, 2018 by jeffreywhunter