The builtin powerdown script can easily hang preventing a shutdown


103 posts in this topic Last Reply

Recommended Posts

I'm marking this as solved because 6.2.1 does a very nice shutdown on my test server.  Good job LT!  Thank you for getting this fixed.

Good to hear that!  Sorry this work should have been done a long time ago.

There may still be issues shutting down VM's within the timeout.  That will be addressed better in subsequent releases.

 

Tom, sorry to have been so pushy about this, but you are right it has needed some attention for a long time.

 

I will now turn my attention to having you do more logging and at least save the diagnostics when shutting down.

 

This would do what I'd like to see:

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown:  Local system shutdown script.
#

# capture the system diagnostics
logger "Capture diagnostics to /boot/logs"
echo "Capture diagnostics to /boot/logs"
rm -f /boot/logs/*diagnostics*.zip
/usr/local/sbin/diagnostics

# Invoke any custom 'stop' script
if [ -f /boot/config/stop ]; then
  echo "Starting stop script"
  fromdos </boot/config/stop >/var/tmp/stop
  chmod +x /var/tmp/stop
  /var/tmp/stop
fi

# Get time-out setting
/usr/bin/fromdos </boot/config/ident.cfg >/var/tmp/ident.cfg
source /var/tmp/ident.cfg
TIMEOUT=${shutdownTimeout:=60}
            
# Find the port emhttp is listening at.
PORT=$(lsof -i -P -sTCP:LISTEN|grep -Pom1 '^emhttp.*:\K\d+')

# (Try) to stop the array
echo "Waiting $TIMEOUT seconds for graceful shutdown..."
if ! timeout $TIMEOUT /usr/bin/wget -q -O - localhost:$PORT/update.htm?cmdStop=apply >/dev/null ;then
  echo "Forcing shutdown..."
fi

# Shutdown cpuload daemon
if [ -x /etc/rc.d/rc.cpuload ]; then
  echo "Stopping cpuload daemon: /etc/rc.d/rc.cpuload"
  /etc/rc.d/rc.cpuload stop
fi

 

Sure we can do that.  I've had some concern in the past about writing to the flash unnecessarily but that's probably not a big concern these days.

Link to post
  • Replies 102
  • Created
  • Last Reply

Top Posters In This Topic

I'm marking this as solved because 6.2.1 does a very nice shutdown on my test server.  Good job LT!  Thank you for getting this fixed.

Good to hear that!  Sorry this work should have been done a long time ago.

There may still be issues shutting down VM's within the timeout.  That will be addressed better in subsequent releases.

 

Tom, sorry to have been so pushy about this, but you are right it has needed some attention for a long time.

 

I will now turn my attention to having you do more logging and at least save the diagnostics when shutting down.

 

This would do what I'd like to see:

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown:  Local system shutdown script.
#

# capture the system diagnostics
logger "Capture diagnostics to /boot/logs"
echo "Capture diagnostics to /boot/logs"
rm -f /boot/logs/*diagnostics*.zip
/usr/local/sbin/diagnostics

# Invoke any custom 'stop' script
if [ -f /boot/config/stop ]; then
  echo "Starting stop script"
  fromdos </boot/config/stop >/var/tmp/stop
  chmod +x /var/tmp/stop
  /var/tmp/stop
fi

# Get time-out setting
/usr/bin/fromdos </boot/config/ident.cfg >/var/tmp/ident.cfg
source /var/tmp/ident.cfg
TIMEOUT=${shutdownTimeout:=60}
            
# Find the port emhttp is listening at.
PORT=$(lsof -i -P -sTCP:LISTEN|grep -Pom1 '^emhttp.*:\K\d+')

# (Try) to stop the array
echo "Waiting $TIMEOUT seconds for graceful shutdown..."
if ! timeout $TIMEOUT /usr/bin/wget -q -O - localhost:$PORT/update.htm?cmdStop=apply >/dev/null ;then
  echo "Forcing shutdown..."
fi

# Shutdown cpuload daemon
if [ -x /etc/rc.d/rc.cpuload ]; then
  echo "Stopping cpuload daemon: /etc/rc.d/rc.cpuload"
  /etc/rc.d/rc.cpuload stop
fi

 

Sure we can do that.  I've had some concern in the past about writing to the flash unnecessarily but that's probably not a big concern these days.

 

Thank you.  That should help with troubleshooting.

 

I understand about your concern about writing to the flash, but I don't believe it's as much an issue these days as in the past.  I'd like to have diagnostics when a person shuts down so we can help them troubleshoot issues.  You know all too well how hard it can be these days, especially with those that insist on pushing the envelope.

Link to post

Updated to 6.2.1, crossed my fingers and removed the PowerDown plugin ... spun down all disks (so the shutdown process would have to spin them up); and then tried my trusty little PLink shutdown script ==> and the server dutifully shut down  :)

 

Not a comprehensive test ... but it sure looks like this is going to do the trick.

 

Link to post

Does the server need to be restarted once Powerdown is removed from the Plugin's page (v6.2.1)?

Thanks

 

Good question.  I don't think so ... I'd think there'd be an indication on the server about needing a reboot (as there is, for example, if you update UnRAID).  But it'd be nice to have confirmation of that.

 

Link to post

I'm marking this as solved because 6.2.1 does a very nice shutdown on my test server.  Good job LT!  Thank you for getting this fixed.

Good to hear that!  Sorry this work should have been done a long time ago.

There may still be issues shutting down VM's within the timeout.  That will be addressed better in subsequent releases.

 

Tom, sorry to have been so pushy about this, but you are right it has needed some attention for a long time.

 

I will now turn my attention to having you do more logging and at least save the diagnostics when shutting down.

 

This would do what I'd like to see:

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown:  Local system shutdown script.
#

# capture the system diagnostics
logger "Capture diagnostics to /boot/logs"
echo "Capture diagnostics to /boot/logs"
rm -f /boot/logs/*diagnostics*.zip
/usr/local/sbin/diagnostics

# Invoke any custom 'stop' script
if [ -f /boot/config/stop ]; then
  echo "Starting stop script"
  fromdos </boot/config/stop >/var/tmp/stop
  chmod +x /var/tmp/stop
  /var/tmp/stop
fi

# Get time-out setting
/usr/bin/fromdos </boot/config/ident.cfg >/var/tmp/ident.cfg
source /var/tmp/ident.cfg
TIMEOUT=${shutdownTimeout:=60}
            
# Find the port emhttp is listening at.
PORT=$(lsof -i -P -sTCP:LISTEN|grep -Pom1 '^emhttp.*:\K\d+')

# (Try) to stop the array
echo "Waiting $TIMEOUT seconds for graceful shutdown..."
if ! timeout $TIMEOUT /usr/bin/wget -q -O - localhost:$PORT/update.htm?cmdStop=apply >/dev/null ;then
  echo "Forcing shutdown..."
fi

# Shutdown cpuload daemon
if [ -x /etc/rc.d/rc.cpuload ]; then
  echo "Stopping cpuload daemon: /etc/rc.d/rc.cpuload"
  /etc/rc.d/rc.cpuload stop
fi

 

Sure we can do that.  I've had some concern in the past about writing to the flash unnecessarily but that's probably not a big concern these days.

 

Thank you.  That should help with troubleshooting.

 

I understand about your concern about writing to the flash, but I don't believe it's as much an issue these days as in the past.  I'd like to have diagnostics when a person shuts down so we can help them troubleshoot issues.  You know all too well how hard it can be these days, especially with those that insist on pushing the envelope.

 

Alternatively you could run diagnostics only when there no proper shutdown (=timeout expired)

 

Link to post

Does the server need to be restarted once Powerdown is removed from the Plugin's page (v6.2.1)?

Thanks

 

I would.  The plugin does mess with some of the system calls for things like Ctrl-Alt-Del that would only be made right with a reboot.

Link to post

Tom,

 

I had a case yesterday on my main server where I had a forced shutdown.  I did end up getting a good shutdown.  I have a lot of plugins, a few Dockers, and a Windows VM.  I don't think I am pushing the envelope.

 

syslog and diagnostics attached.

 

As you can see the array drives did not unmount because they were busy.

 

EDIT: I don't know if this is helpful, but the powerdown plugin did one last cleanup to be sure there were no remaining active pids on the array drives before unmounting them:

echo "Killing active pids on the array drives..."
logger "Killing active pids on the array drives..."
/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

It skipped skips /mnt/disks/ because this is the UD mount point.

syslog.txt

mediaserver-diagnostics-20161005-0636.zip

Link to post

Tom,

 

I had a case yesterday on my main server where I had a forced shutdown.  I did end up getting a good shutdown.  I have a lot of plugins, a few Dockers, and a Windows VM.  I don't think I am pushing the envelope.

 

syslog and diagnostics attached.

 

As you can see the array drives did not unmount because they were busy.

 

EDIT: I don't know if this is helpful, but the powerdown plugin did one last cleanup to be sure there were no remaining active pids on the array drives before unmounting them:

echo "Killing active pids on the array drives..."
logger "Killing active pids on the array drives..."
/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

It skipped /mnt/disks/ because this is the UD mount point.

 

What process was holding an open file descriptor on /mnt/user/... ?

 

Also, what do you think of bonienl's idea of only generating diagnostics.zip when there is a forced shutdown (meaning timeout expired waiting for cmdStop)?

Link to post

dlandon => You said this in the 6.2.1 thread:

 

...

If you use either /sbin/powerdown or /usr/local/sbin/powerdown in any scripts, you need to change them to the newer methods:

- Change 'powerdown' to '/sbin/poweroff'

- Change 'powerdown -r' to '/sbin/reboot'

 

My current shutdown script within Windows is this line:

 

S:\Downloads\Putty\Plink.exe -ssh -pw <password> root@<servername> powerdown

 

I presume I should change the "powerdown" to "poweroff".  Is that right?  ... and is there any change to the syntax?

 

Link to post

Tom,

 

I had a case yesterday on my main server where I had a forced shutdown.  I did end up getting a good shutdown.  I have a lot of plugins, a few Dockers, and a Windows VM.  I don't think I am pushing the envelope.

 

syslog and diagnostics attached.

 

As you can see the array drives did not unmount because they were busy.

 

EDIT: I don't know if this is helpful, but the powerdown plugin did one last cleanup to be sure there were no remaining active pids on the array drives before unmounting them:

echo "Killing active pids on the array drives..."
logger "Killing active pids on the array drives..."
/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

It skipped /mnt/disks/ because this is the UD mount point.

 

What process was holding an open file descriptor on /mnt/user/... ?

 

I don't have any idea.  I just did a normal shutdown using the Dynamix Buttons plugin.  I believe that plugin calls "powerdown -r" on reboot.  How can I get you more information when this happens?  Is there any logging on why the shutdown was forced?  Should there be?

 

Can you add something like I suggested to force active pids off array devices and log the results so we can see in the log what is holding up the unmounts?  Trying to track this down is going to be difficult without some information logged to see what and where this is happening.

 

FYI: It looks like Dockers and my VM did shutdown properly.  I would guess it's a timing thing with plugins stopping.

 

Also, what do you think of bonienl's idea of only generating diagnostics.zip when there is a forced shutdown (meaning timeout expired waiting for cmdStop)?

 

That would work just fine.  It's probably only needed when the shutdown is forced, meaning that something has gone wrong.

Link to post

dlandon => You said this in the 6.2.1 thread:

 

...

If you use either /sbin/powerdown or /usr/local/sbin/powerdown in any scripts, you need to change them to the newer methods:

- Change 'powerdown' to '/sbin/poweroff'

- Change 'powerdown -r' to '/sbin/reboot'

 

My current shutdown script within Windows is this line:

 

S:\Downloads\Putty\Plink.exe -ssh -pw <password> root@<servername> powerdown

 

I presume I should change the "powerdown" to "poweroff".  Is that right?

 

Yes.  That's what I said.

 

... and is there any change to the syntax?

 

No.

Link to post

Trying to track this down is going to be difficult without some information logged to see what and where this is happening.

 

If you bring your server all the way back up, then go to Main and click 'Stop' you should see the same 'hanging' right?  Then you should be able to use fuser or ps to figure out what's not cleanly shutting down.

Link to post

Trying to track this down is going to be difficult without some information logged to see what and where this is happening.

 

If you bring your server all the way back up, then go to Main and click 'Stop' you should see the same 'hanging' right?  Then you should be able to use fuser or ps to figure out what's not cleanly shutting down.

 

Unfortunately it is very sporadic and hard to reproduce.  I will try to stop the array and see if it hangs and get you more information if I can.

Link to post

I can't believe it, but I did get it to hang on the second try.  Ok, so here is what I have:

 

This command:

/usr/bin/fuser -mv /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

produced the following in the log:

Oct  5 13:00:25 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:25 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:25 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:25 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6

 

This command let the shutdown continue:

/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

and produced the following in the log:

Oct  5 13:00:32 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:32 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:32 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:32 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6
Oct  5 13:00:33 MediaServer emhttp: shcmd (9134): set -o pipefail ; umount /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9135): rmdir /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9136): rm -f /boot/config/plugins/dynamix/mover.cron
Oct  5 13:00:33 MediaServer emhttp: shcmd (9137): /usr/local/sbin/update_cron &> /dev/null
Oct  5 13:00:33 MediaServer emhttp: Unmounting disks...
Oct  5 13:00:33 MediaServer emhttp: shcmd (9138): umount /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer kernel: XFS (md1): Unmounting Filesystem
Oct  5 13:00:33 MediaServer emhttp: shcmd (9139): rmdir /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9140): umount /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md2): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9141): rmdir /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9142): umount /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md3): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9143): rmdir /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9144): umount /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md4): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9145): rmdir /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9146): umount /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md5): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9147): rmdir /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9148): umount /mnt/disk6 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md6): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9149): rmdir /mnt/disk6 |& logger
Oct  5 13:00:35 MediaServer emhttp: shcmd (9150): umount /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: XFS (sdi1): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9151): rmdir /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: mdcmd (53): stop 
Oct  5 13:00:35 MediaServer kernel: md1: stopping
Oct  5 13:00:35 MediaServer kernel: md2: stopping
Oct  5 13:00:35 MediaServer kernel: md3: stopping
Oct  5 13:00:35 MediaServer kernel: md4: stopping
Oct  5 13:00:35 MediaServer kernel: md5: stopping
Oct  5 13:00:35 MediaServer kernel: md6: stopping
Oct  5 13:00:35 MediaServer emhttp: shcmd (9152): rmmod md-mod |& logger
Oct  5 13:00:35 MediaServer kernel: md: unRAID driver removed

 

Hope this is helpful.

 

EDIT: All shares were still available.  I attached the complete log in case you want to see it.

mediaserver-syslog-20161005-1333.zip

Link to post

I know this might be unhelpful without logs and testing.  But the main reason I previously had hangs while using the built in power/stop button in the gui was always on unmounting disks.  This is/was across many different machines and unraid versions etc. I would assume (and a few times proved) that it was due to something being open,  ie mc telnet session. I assume unraid didn't want to kill that connection in case it was during a write? When a machine would get hung in this state the powerdown plugin would be a solution to kill the pid and move on with a safe shutdown. 

 

Moving forward,  will unraid be able to get past any unmounting hangs during the stopping of the array?

 

Sent using Tapatalk

 

Link to post

I think we can safely assume users will not be writing to the array while stopping it. But since users are users, we should have an italic text at the bottom of the Stop button saying "Please make sure you are not using the array via a telnet/ssh session. They will be killed if the timeout expires." warning text.

 

I am running 6.2.1 now, and I must say I'm OK with the current implementation. Though there is a long delay between the stop button press and the array stopping completely. Also, it would be a nice idea to bring back the power off and reboot buttons while the array is running. The idea is that if we press power off/reboot button, it would automatically stop the array and reboot for us, and we don't have to stick around waiting for the array to stop, then hitting the power off/reboot button in the webUI.

 

Just a suggestion. I'm still happy with what the admins choose to keep/discard.

Link to post

I can't believe it, but I did get it to hang on the second try.  Ok, so here is what I have:

 

This command:

/usr/bin/fuser -mv /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

produced the following in the log:

Oct  5 13:00:25 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:25 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:25 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:25 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6

 

This command let the shutdown continue:

/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

and produced the following in the log:

Oct  5 13:00:32 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:32 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:32 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:32 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6
Oct  5 13:00:33 MediaServer emhttp: shcmd (9134): set -o pipefail ; umount /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9135): rmdir /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9136): rm -f /boot/config/plugins/dynamix/mover.cron
Oct  5 13:00:33 MediaServer emhttp: shcmd (9137): /usr/local/sbin/update_cron &> /dev/null
Oct  5 13:00:33 MediaServer emhttp: Unmounting disks...
Oct  5 13:00:33 MediaServer emhttp: shcmd (9138): umount /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer kernel: XFS (md1): Unmounting Filesystem
Oct  5 13:00:33 MediaServer emhttp: shcmd (9139): rmdir /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9140): umount /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md2): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9141): rmdir /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9142): umount /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md3): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9143): rmdir /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9144): umount /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md4): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9145): rmdir /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9146): umount /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md5): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9147): rmdir /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9148): umount /mnt/disk6 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md6): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9149): rmdir /mnt/disk6 |& logger
Oct  5 13:00:35 MediaServer emhttp: shcmd (9150): umount /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: XFS (sdi1): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9151): rmdir /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: mdcmd (53): stop 
Oct  5 13:00:35 MediaServer kernel: md1: stopping
Oct  5 13:00:35 MediaServer kernel: md2: stopping
Oct  5 13:00:35 MediaServer kernel: md3: stopping
Oct  5 13:00:35 MediaServer kernel: md4: stopping
Oct  5 13:00:35 MediaServer kernel: md5: stopping
Oct  5 13:00:35 MediaServer kernel: md6: stopping
Oct  5 13:00:35 MediaServer emhttp: shcmd (9152): rmmod md-mod |& logger
Oct  5 13:00:35 MediaServer kernel: md: unRAID driver removed

 

Hope this is helpful.

 

EDIT: All shares were still available.  I attached the complete log in case you want to see it.

 

Got some time to look at this issue in more detail... From the syslog:

 

This is where it looks like shutdown was started:

Oct  4 21:58:43 MediaServer emhttp: Stopping services...

 

Everything looks ok so far, then after initiating docker container shutdown we see:

Oct  4 21:58:58 MediaServer kernel: zmdc.pl[10613]: segfault at 18 ip 00002ab0edd304a9 sp 00007ffc406e9950 error 4 in libc-2.19.so[2ab0edcb0000+1bb000]

 

I don't think that has anything to do with array Stop timeout, I just thought I'd point that out.  Moving on...

 

Here is where Samba is stopped:

Oct  4 21:59:10 MediaServer emhttp: shcmd (1693): /etc/rc.d/rc.samba stop |& logger

 

But a few lines later we see:

Oct  4 21:59:12 MediaServer emhttp: Recycle Bin stopped...
Oct  4 21:59:12 MediaServer Recycle Bin: Recycle Bin stopped

 

I'm assuming this is a plugin using the samba recycle bin feature?  If so, something is not right because when

/etc/rc.d/rc.samba stop

is executed, it should have completely killed the nmbd and all smbd processes.  But the reason unmount is hanging later is because there is still an smbd process holding the /mnt/user mount point.  Hence either rc.samba stop did not actually stop samba (because of the plugin), or later tying to stop the recycle bin plugin actually restarted a smbd process.

 

My suggestion is to remove plugins until you get a clean shutdown and then figure out what's wrong with the last removed plugin  :P

 

Link to post

Moving forward,  will unraid be able to get past any unmounting hangs during the stopping of the array?

 

The array Stop operation, commonly invoked by clicking 'Stop' on the Main page is a distinct operation that is also invoked by the shutdown process.  The Stop operation itself does not have a time-out.  The main reason a Stop might hang is due to an open file on a mounted file system.  You correctly point out the most common way this happens is someone holding command shell open, eg, 'mc'.  In this case it would probably be ok to just kill the process.  However there are other reasons.  For example, maybe someone has a loopback-mounted file which is located on the array/cache.  In this case it will not show up in 'fuser', but if we force a Stop it could result in data loss inside that loobback.  Hence when invoked manually (by clicking Stop button) unRAID takes the safe approach and if it can't unmount array/cache file sytems it sits in a loop until you, the user, correct what's wrong.

 

The case of poweroff is similar but with a key difference.  Sure poweroff can be invoked manually but it is also tied into the apcupsd feature.  If power is lost and server is running on battery, if apcupsd says "poweroff" we have to power-off right now!  In this case if the array Stop operation is hanging in an unmount loop, well too bad, after 60 seconds (or whatever you set), we are going to kill it and continue with the shutdown.  As shutdown proceeds it will forcibly kill all the processes in the system, forcibly unmount all the file systems, stop the md/unraid driver, and then finally turn off power.

Link to post

Also, it would be a nice idea to bring back the power off and reboot buttons while the array is running. The idea is that if we press power off/reboot button, it would automatically stop the array and reboot for us, and we don't have to stick around waiting for the array to stop, then hitting the power off/reboot button in the webUI.

 

Yes we want to add this enhancement.  Hopefully it will get into 6.3.

Link to post

I can't believe it, but I did get it to hang on the second try.  Ok, so here is what I have:

 

This command:

/usr/bin/fuser -mv /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

produced the following in the log:

Oct  5 13:00:25 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:25 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:25 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:25 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:25 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:25 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:25 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:25 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:25 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:25 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:25 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:25 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:25 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6

 

This command let the shutdown continue:

/usr/bin/fuser -mvk /mnt/user/* /mnt/disk*[!s] /mnt/cache /dev/md* 2>&1 | logger

 

and produced the following in the log:

Oct  5 13:00:32 MediaServer root:                      USER        PID ACCESS COMMAND
Oct  5 13:00:32 MediaServer root: /mnt/user/AV Media:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Computer Backups:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/DVD Movies:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Movies:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Music:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/NetDrive:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Pictures:  root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Public:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Software Development:
Oct  5 13:00:32 MediaServer root:                      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Users:     root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/Videos:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/appdata:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/docker:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/domains:   root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/iTunes:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/isos:      root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/user/system:    root     kernel mount /mnt/user
Oct  5 13:00:32 MediaServer root:                      root      27376 f.c.. smbd
Oct  5 13:00:32 MediaServer root:                      root      27725 f.c.. smbd
Oct  5 13:00:32 MediaServer root: /mnt/disk1:          root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /mnt/disk2:          root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /mnt/disk3:          root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /mnt/disk4:          root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /mnt/disk5:          root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /mnt/disk6:          root     kernel mount /mnt/disk6
Oct  5 13:00:32 MediaServer root: /mnt/cache:          root     kernel mount /mnt/cache
Oct  5 13:00:32 MediaServer root: /dev/md1:            root     kernel mount /mnt/disk1
Oct  5 13:00:32 MediaServer root: /dev/md2:            root     kernel mount /mnt/disk2
Oct  5 13:00:32 MediaServer root: /dev/md3:            root     kernel mount /mnt/disk3
Oct  5 13:00:32 MediaServer root: /dev/md4:            root     kernel mount /mnt/disk4
Oct  5 13:00:32 MediaServer root: /dev/md5:            root     kernel mount /mnt/disk5
Oct  5 13:00:32 MediaServer root: /dev/md6:            root     kernel mount /mnt/disk6
Oct  5 13:00:33 MediaServer emhttp: shcmd (9134): set -o pipefail ; umount /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9135): rmdir /mnt/user |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9136): rm -f /boot/config/plugins/dynamix/mover.cron
Oct  5 13:00:33 MediaServer emhttp: shcmd (9137): /usr/local/sbin/update_cron &> /dev/null
Oct  5 13:00:33 MediaServer emhttp: Unmounting disks...
Oct  5 13:00:33 MediaServer emhttp: shcmd (9138): umount /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer kernel: XFS (md1): Unmounting Filesystem
Oct  5 13:00:33 MediaServer emhttp: shcmd (9139): rmdir /mnt/disk1 |& logger
Oct  5 13:00:33 MediaServer emhttp: shcmd (9140): umount /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md2): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9141): rmdir /mnt/disk2 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9142): umount /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md3): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9143): rmdir /mnt/disk3 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9144): umount /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md4): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9145): rmdir /mnt/disk4 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9146): umount /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md5): Unmounting Filesystem
Oct  5 13:00:34 MediaServer emhttp: shcmd (9147): rmdir /mnt/disk5 |& logger
Oct  5 13:00:34 MediaServer emhttp: shcmd (9148): umount /mnt/disk6 |& logger
Oct  5 13:00:34 MediaServer kernel: XFS (md6): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9149): rmdir /mnt/disk6 |& logger
Oct  5 13:00:35 MediaServer emhttp: shcmd (9150): umount /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: XFS (sdi1): Unmounting Filesystem
Oct  5 13:00:35 MediaServer emhttp: shcmd (9151): rmdir /mnt/cache |& logger
Oct  5 13:00:35 MediaServer kernel: mdcmd (53): stop 
Oct  5 13:00:35 MediaServer kernel: md1: stopping
Oct  5 13:00:35 MediaServer kernel: md2: stopping
Oct  5 13:00:35 MediaServer kernel: md3: stopping
Oct  5 13:00:35 MediaServer kernel: md4: stopping
Oct  5 13:00:35 MediaServer kernel: md5: stopping
Oct  5 13:00:35 MediaServer kernel: md6: stopping
Oct  5 13:00:35 MediaServer emhttp: shcmd (9152): rmmod md-mod |& logger
Oct  5 13:00:35 MediaServer kernel: md: unRAID driver removed

 

Hope this is helpful.

 

EDIT: All shares were still available.  I attached the complete log in case you want to see it.

 

Got some time to look at this issue in more detail... From the syslog:

 

This is where it looks like shutdown was started:

Oct  4 21:58:43 MediaServer emhttp: Stopping services...

 

Everything looks ok so far, then after initiating docker container shutdown we see:

Oct  4 21:58:58 MediaServer kernel: zmdc.pl[10613]: segfault at 18 ip 00002ab0edd304a9 sp 00007ffc406e9950 error 4 in libc-2.19.so[2ab0edcb0000+1bb000]

 

I don't think that has anything to do with array Stop timeout, I just thought I'd point that out.  Moving on...

 

Here is where Samba is stopped:

Oct  4 21:59:10 MediaServer emhttp: shcmd (1693): /etc/rc.d/rc.samba stop |& logger

 

But a few lines later we see:

Oct  4 21:59:12 MediaServer emhttp: Recycle Bin stopped...
Oct  4 21:59:12 MediaServer Recycle Bin: Recycle Bin stopped

 

I'm assuming this is a plugin using the samba recycle bin feature?  If so, something is not right because when

/etc/rc.d/rc.samba stop

is executed, it should have completely killed the nmbd and all smbd processes.  But the reason unmount is hanging later is because there is still an smbd process holding the /mnt/user mount point.  Hence either rc.samba stop did not actually stop samba (because of the plugin), or later tying to stop the recycle bin plugin actually restarted a smbd process.

 

My suggestion is to remove plugins until you get a clean shutdown and then figure out what's wrong with the last removed plugin  :P

 

Ok.  This is good information and I'll take a look at the Recycle Bin plugin.  It does make changes to samba and it may be restarting samba.

Link to post

Moving forward,  will unraid be able to get past any unmounting hangs during the stopping of the array?

 

The array Stop operation, commonly invoked by clicking 'Stop' on the Main page is a distinct operation that is also invoked by the shutdown process.  The Stop operation itself does not have a time-out.  The main reason a Stop might hang is due to an open file on a mounted file system.  You correctly point out the most common way this happens is someone holding command shell open, eg, 'mc'.  In this case it would probably be ok to just kill the process.  However there are other reasons.  For example, maybe someone has a loopback-mounted file which is located on the array/cache.  In this case it will not show up in 'fuser', but if we force a Stop it could result in data loss inside that loobback.  Hence when invoked manually (by clicking Stop button) unRAID takes the safe approach and if it can't unmount array/cache file sytems it sits in a loop until you, the user, correct what's wrong.

 

The problem is that the gui is unusable while it's waiting for the array to stop, so you can't switch over to the Open Files plugin to see what is causing the problem.

 

I feel like basic functionality of the Open Files plugin should be built into the Array Operation tab, so it can warn you about things like an open bash shell before you try stopping the array.

Link to post

For example, maybe someone has a loopback-mounted file which is located on the array/cache.  In this case it will not show up in 'fuser', but if we force a Stop it could result in data loss inside that loobback.

 

I've certainly done this in the past and been scratching my head trying to work out what the problem was as fuser showed no open file.

 

Running 'mount' to list the mounted file systems made it all clear and I was able to unmount the loopback and the system proceeded to shutdown correctly.

 

 

 

 

Link to post

From a general use perspective, the problem is that most of us know how to troubleshoot these issues -- i.e. if there's an issue with hanging, we can repeat the process and look at open files and mount points to see what might be causing a problem ... but the general user isn't likely to do that => they'll just be frustrated that when they try to power off the server it doesn't work.

 

The PowerDown plugin pretty much resolved this issue over the past few years; but it is indeed VERY nice to see that the built-in function in UnRAID is now being updated so that plugin is no longer needed.  I'm confident that between Tom & dlandon's dialogue in this thread the need for the plugin will be completely gone by 6.2.2  :)

Indeed, 6.2.1 resolves it for most folks ... a few more "tweaks" and it should be "bullet proof"  :)  [Yes, I know there will always be someone who attacks it with a Kevlar-piercing round  :) ]

 

 

Link to post
Guest
This topic is now closed to further replies.